Programming with Peripherals
In this article we’re going to at programming with and for peripherals, including keyboard, mouse, as gesture devices such as Leap Motion and Windows Kinect as well as devices such as pens and microphones.
To get the inside track, I spoke with Earl Flormata, a colleague at Big Box Apps. Here’s what he had to say: With voice, some people are better at vocalizing commands. It’s rare, but I actually met a programmer who likes to say his code, rather than write it or type it. One such program for that is Dragon Naturally speaking. If you take the time to let the program learn your voice, it begins to understand you at a better rate.
As you train the program, it can learn accents, colloquialisms, whatever you want it to do. As an example, I have a programming friend who mapped every swear word to an undo, so if he made a mistake and cursed, it would undo the previous passage.
For pens, there is a writing pad that goes with it. It’s a normal pen, but what it writes is digitized. The term for it is object character recognition (OCR). This is also seen with scanners where the print or writing on a document can be digitized and turned into recognizable text. In this case we’re talking about a real tactile pen on paper. Check into Lightscribe for doing that.
For the most part, coding is a lot of typing and logical thinking. Some people are unable to code that way. They need to write it down, figure it out then they can type it in.
With Windows Kinect, a great way of thinking of how to use it is with that Tom Cruise movie, Minority Report. With Kinect, it’s possible to program the gestures of your hands to switch between tabs and do certain tasks. So for example, if you were coding in HTML or PHP where you have to look at the browser to see how it works. I could use Kinect to switch back and forth between code or… with a movement of my hand. It could be faster than a mouse and you could use Kinect to switch between things.
A Kinect is two cameras set up which recognizes people’s faces, etc. You could use it to log in because it will recognize who you are. You could use it to recognize standing up and moving around. Another example is where you blink your eyes to create a mouse click.
Leap motion it is a band you could wear around your wrist or elbow. What this does is interact with your muscles so if you flex your muscles you can get it to do certain things, such as create a rotation, move left, right and manipulate the mouse as well. It is a newer technology.
Programming for Peripherals
What’s really important is to discover what data the peripheral is giving you. In the case of sound and a sound file, the software has to be able to recognize what you’re saying and then come back and perform an action. The key to making that work is to be able to taking that data and sending it out to the server.
Here’s the issue, at some point when capturing sound, it has to be sent up to the server to be calculated, to find out what it says. An example is Shazam. This is an app you can have on your phone so when you hear a song you like, you push a button, it takes the song information, uploads it to the Internet then finds the song. Now some songs will take longer to figure out than others and some songs don’t have a definitive structure within them. It can figure out the beat structure, lyrics and look for songs that have a matching structure.
Nuance has it to the point where they will insure your voice print for up to $100,000.00. They guarantee that no one else will have a similar voice print. Even identical twins have different voice prints.
Programming for a pen is pretty easy. If you think about what writing is, if I draw a line from point ‘A’ to point ‘B,’ all you’re doing is calculating the coordinates. In addition to the line, you’re also calculating for when the pen touches the paper. Color change is easy to do. The tough part is the hardware. Some hardware will have more sensor points and it can tell you how hard the person is pushing down on the pen. So pressure sensitivity is the thing. If there’s no pressure sensitivity it doesn’t count with your signature for a lot of banking instruments. Some things will let you sign with the mouse while others won’t unless there’s pressure sensitivity built into the device which you use to write your signature.
Nathan: One thing though, trying to write with a mouse is akin to trying to write with a baseball bat. It’s really tough.
In terms of programming for Leap Motion, the function is already built-in and you can use it to recognize faces and gestures but there’s only so much it can recognize. You just have to be able to map that to a function. As an example if I move my hand to the left, that would be ‘undo’ and if I move my hand to the right it’s ‘redo.’
With the game Dance Dance Revolution people use a giant dancing pad and there is an up/down, left/right arrow and you have to step on that arrow according to the image on the screen. With Kinect, it goes beyond that and can recognize the gesture, as well. You can use the hand gestures in the air for other dance moves.
If you were to make a game, you could do one on boxing, where you stick your left hand out, your character would stick his left hand out. You can create some depth perception because there are two cameras and could a create type of 3D data.