A while ago I built a computer input device using a laser pointer and regular usb web camera.
It was a pretty simple setup, and I used a lot of existing tools as a jumping off point. Here’s a writeup of my work and details for how to replicate it and what I learned.
First, a video overview:
At a minimum:
Optionally:
Technically speaking, the laser is completely optional. In fact, during testing I just had a desktop computer with the camera pointed at a sheet of paper taped to a wall, and I drew with the laser pointer on that sheet of paper and used that as an input device. With the projector, you can turn the setup into a more direct input, as your point translates directly to a point on a screen. But that’s just a bonus. This can be done without the projector.
Take the camera, point it at something. Shine the laser inside the area where the camera can see. That’s it in a nutshell. However, there are some additional considerations.
First, the more direct the camera, the more accurate it will be. If the camera is off to the side, it will see a skewed wall, and because one side is closer than the other, it’s impossible to focus perfectly, and one side of the wall will be more precise than the far side of the wall. Having the camera point as directly as possible at the surface is the best option.
Second, the distance to the surface matters. A camera that is too far from the surface may not be able to see a really small laser point. A camera that is too close will see a very large laser point and will not have great resolution. It’s a tradeoff, and you just have to experiment to find the best distance.
Third, if using a projector, the camera should be able to see slightly more than the projected area. A border of a few inches up to a foot is preferable, as this space can actually be used for input even if it’s not in the projected space.
Fourth, it’s important to control light levels. If there are sources of light in the view of the camera, such as a lamp or window, then it is very likely the algorithm will see those points as above the threshold, and will try to consider them part of the laser pointer (remember white light is made up of red, green, and blue, so it will still be above the red threshold). Also, if using a projector, the laser pointer has to be brighter than the brightest the projector can get, and the threshold has to be set so that the projector itself isn’t bright enough to go over the threshold. And the ambient light in the room can’t be so bright that the threshold has to be really high and thus the laser pointer isn’t recognized. Again, there are a lot of tradeoffs with the light levels in a room.
I wrote my software in Java. There are two libraries that I depended on heavily:
The JAI library is not entirely essential, as you could decide not to translate your coordinates, or you could perform your affine transform math to do it and eschew the large library that will go mostly unused. The neat thing about this transform, though, is that it allows for the camera to be anywhere, and as long as it can see the desired area, it will take care of transforming to the correct coordinates. This is very convenient.
The JMF library exists for Windows, Linux, and Mac. I was able to get it working in Windows, but wasn’t able to get it completely working in Linux (Ubuntu Jaunty as of this writing), and I don’t have a Mac to test on.
The basic theory behind the project is the following; a laser pointer shines on a surface. The web camera is looking at that surface. Software running on a computer is analyzing each frame of that camera image and looking for the laser pointer. Once it finds that pointer, it converts the camera coordinates of that point into screen coordinates and fires an event to any piece of software that is listening. That listening software can then do something with that event. The simplest example is a mouse emulator, which merely moves the mouse to the correct coordinates based on the location of the laser.
To implement this, I have the JMF library looking at each frame. I used this frameaccess.java example code as a starting point. When looking at each frame, I only look at the 320×240 frame, and specifically only at the red value. Each pixel has a value for red, green, and blue, but since this is a red laser I’m looking at, I don’t really care about anything but red. I traverse the entire frame and create a list of any pixels above a certain threshold value. These are the brightest pixels in the frame and very likely a laser pointer. Then I average the locations of these points and come up with a single number. This is very important, and I’ll describe some of the effects that this has later. I take this point and perform the affine transform to convert it to screen coordinates. Then I have certain events that get fired depending on some specific things:
For most of these events, the raw camera coordinates and the transformed coordinates are passed to the listeners. The listeners then do whatever they want with this information.
Calibration is really only necessary if you are using the coordinate transforms. Essentially, the calibration process consists of identifying four points and mapping camera coordinates to the other coordinates. I wrote a small application that shows a blank screen and prompts the user to put the laser point at each of the prompted circles, giving the system a mapping at known locations. This writes the data to a configuration file which is used by all other applications. As long as the camera and projector don’t move, calibration does not need to be done again.
Here is a video of the calibration process.
Here is the laser camera code (3.7mb). It includes the JAI library, the base laser application, the calibrator, and an example application that just acts as a mouse emulator.
Below are a couple snippets of the important stuff.
This first part is the code used to parse each frame and find the laser point, then fire the appropriate events.
/**
* Callback to access individual video frames. This is where almost all of the work is done.
*/
void accessFrame(Buffer frame) {
/***************************************************************************************/
/********************************Begin Laser Detection Code*****************************/
/***************************************************************************************/
// Go through all the points and set them to an impossible number
for (int i = 0;i<points.length;i++){
points[i].x = -1;
points[i].y = -1;
}
int inc = 0; //set our incrementer to 0
byte[] data = (byte[])frame.getData(); //grab the frame data
for (int i = 0;i<data.length;i+=3){//go through the whole buffer (jumping by three because we only want the red out of the RGB
//if(unsignedByteToInt(data[i+2])>THRESHOLD && unsignedByteToInt(data[i+1])<LOWERTHRESHOLD && unsignedByteToInt(data[i+0])<LOWERTHRESHOLD && inc<points.length){//if we are above the threshold and our incrementer is below the maximum number of points
if(unsignedByteToInt(data[i+2])>THRESHOLD && inc<points.length){//if we are above the threshold and our incrementer is below the maximum number of points
points[inc].x = (i%(3*CAMERASIZEX))/3; //set the x value to that coordinate
points[inc].y = i/(3*CAMERASIZEX); //set the y value to the right line
inc++;
}
}
//calculate the average of the points we found
ave.x = 0;
ave.y = 0;
for (int i=0;i<inc;i++){
if (points[i].x!=-1){
ave.x+=points[i].x;
}
if (points[i].y!=-1){
ave.y+=points[i].y;
}
//System.out.println(points[i].x + "," + points[i].y);
}
//System.out.println("-------------------");
if (inc>3){//if we found enough points that we probably have a laser pointer on the screen
ave.x/=inc;//finish calculating the average
ave.y/=inc;
PerspectiveTransform mytransform = PerspectiveTransform.getQuadToQuad(mapping[0].getX(), mapping[0].getY(),
mapping[1].getX(), mapping[1].getY(), mapping[2].getX(), mapping[2].getY(), mapping[3].getX(), mapping[3].getY(),
correct[0].getX(), correct[0].getY(), correct[1].getX(), correct[1].getY(), correct[2].getX(), correct[2].getY(), correct[3].getX(), correct[3].getY());
Point2D result = mytransform.transform(new Point(ave.x,ave.y),null);
in_space = !(result.getX()<0 || result.getY() < 0 || result.getX() > SCREENSIZEX || result.getY() > SCREENSIZEY);
if (!on){
fireLaserOn(new LaserEvent(result, new Point(ave.x, ave.y), last_point, last_raw_point,in_space));
on = true;
}
if (in_space && !last_in_space){
fireLaserEntered(new LaserEvent(result, new Point(ave.x, ave.y), last_point, last_raw_point,true));
}
// System.out.println(result.getX() + "," + result.getY());
// System.out.println(last_point.getX() + "," + last_point.getY());
// System.out.println("----------------------");
if (result.getX()!=last_point.getX() || result.getY()!=last_point.getY()){
fireLaserMoved(new LaserEvent(result, new Point(ave.x, ave.y), last_point, last_raw_point,in_space));
}
else{
fireLaserStable(new LaserEvent(result, new Point(ave.x, ave.y), last_point, last_raw_point,in_space));
}
if (!in_space && last_in_space){
fireLaserExited(new LaserEvent(result, new Point(ave.x, ave.y), last_point, last_raw_point,false));
}
last_time = 0;
last_point = new Point2D.Double(result.getX(), result.getY());
}
else if (last_time==5){//if it's been five frames since we last saw the pointer, then it must have disappeared
if (in_space){
fireLaserExited(new LaserEvent(-1,-1, ave.x, ave.y, (int)last_point.getX(), (int)last_point.getY(), (int)last_raw_point.getX(), (int)last_raw_point.getY(),in_space));
}
fireLaserOff(new LaserEvent(-1,-1, ave.x, ave.y, (int)last_point.getX(), (int)last_point.getY(), (int)last_raw_point.getX(), (int)last_raw_point.getY(),in_space));
on = false;
in_space = false;
}
if (ave.x>0 || ave.y>0 && ave.x<CAMERASIZEX && ave.y<CAMERASIZEY)
fireLaserRaw(new LaserEvent(-1,-1, ave.x, ave.y, -1,-1, (int)last_raw_point.getX(), (int)last_raw_point.getY(),in_space));
last_time++;//increment the last_time. usually it gets set to 0 every frame if the laser is there
last_raw_point = new Point(ave.x,ave.y);//set the last_point no matter what
last_in_space = in_space;
/**************************************************************************************/
/********************************End Laser Detection Code*****************************/
/*************************************************************************************/
}
public int unsignedByteToInt(byte b) {
return (int) b & 0xFF;
}
This next part is pretty standard code for adding event listeners. You can see which laser events are getting passed. I intentionally made it similar to how mouse listeners are used.
Vector<LaserListener> laserListeners = new Vector<LaserListener>();
public void addLaserListener(LaserListener l){
laserListeners.add(l);
}
public void removeLaserListener(LaserListener l){
laserListeners.remove(l);
}
private void fireLaserOn(LaserEvent e){
Enumeration<LaserListener> en = laserListeners.elements();
while(en.hasMoreElements()){
LaserListener l = (LaserListener)en.nextElement();
l.laserOn(e);
}
}
private void fireLaserOff(LaserEvent e){
Enumeration<LaserListener> en = laserListeners.elements();
while(en.hasMoreElements()){
LaserListener l = (LaserListener)en.nextElement();
l.laserOff(e);
}
}
private void fireLaserMoved(LaserEvent e){
Enumeration<LaserListener> en = laserListeners.elements();
while(en.hasMoreElements()){
LaserListener l = (LaserListener)en.nextElement();
l.laserMoved(e);
}
}
private void fireLaserStable(LaserEvent e){
Enumeration<LaserListener> en = laserListeners.elements();
while(en.hasMoreElements()){
LaserListener l = (LaserListener)en.nextElement();
l.laserStable(e);
}
}
private void fireLaserEntered(LaserEvent e){
Enumeration<LaserListener> en = laserListeners.elements();
while(en.hasMoreElements()){
LaserListener l = (LaserListener)en.nextElement();
l.laserEntered(e);
}
}
private void fireLaserExited(LaserEvent e){
Enumeration<LaserListener> en = laserListeners.elements();
while(en.hasMoreElements()){
LaserListener l = (LaserListener)en.nextElement();
l.laserExited(e);
}
}
private void fireLaserRaw(LaserEvent e){
Enumeration<LaserListener> en = laserListeners.elements();
while(en.hasMoreElements()){
LaserListener l = (LaserListener)en.nextElement();
l.laserRaw(e);
}
}
* If you do use this, please send me a note at laser@bobbaddeley.com. It’s always neat to see other people using my stuff.