You are on page 1of 22

Accelerometer Based Hand Action Recognition

Zhe Yang (zy49) Kun Yi (ky237)


High Level Design

Hardware Software Results Conclusion Appendices

We created a wearable game controller that uses accelerometers to acquire action of the hand and then maps an action to an arbitrary keystroke. The types of actions we are trying to recognize should be suitable as input control for video games.

We placed 3 z-axis accelerometers on tips of the thumb, the index finger and the middle finger, and three accelerometers on the back of the hand for x,y, and z acceleration. The Atmega644 microcontroller read the output of the accelerometers and simulates a finite state machine to compute the gesture and the motion of the hand. The gesture and motion information is then transmitted to PC through serial connection, and a Java program is used to read the information and map it to an arbitrary keystroke.

High level design
Some current game controllers such as the Wii remote take use of sensors and are capable of using the player¶s motion as input, to some extent. However, none of the controlling method can take advantage of the expressibilities of natural gestures. As avid gamers, we determined that a hand-action-based controller would be a novel and fun input device.

By observation, we have found that many meaningful actions performed by a hand (e.g. smash, swing or push) can be described by the hand gesture and the movement of the palm. Furthermore, the hand gesture should be independent with the hand movement and orientation, but can only be changed from one to another by movements of fingers. On the other hand, the orientation is only dependent on the movement of the hand. Therefore in our project, we propose a prototype of real time hand action recognition by 8-bit microcontroller using acceleration data from finger tips and back of the hand.

Background Math and Physics
Accelerometers: the accelerometers we used measures acceleration from a capacitive sensing cell (g-cell) which forms two back-to-back capacitors. As shown in Figure 1, when the center plate deflects due to acceleration, the value of capacitors will change and acceleration data can be extracted.

Figure 1: Physical Model of the Accelerometer

Each accelerometer¶s reading consists of dynamic acceleration and static acceleration, namely the gravity. For a z-axis accelerometer placed flat, Figure 2 shows the sign of the reading. The static acceleration is +1g, where as the dynamic acceleration reading is positive if the direction of the acceleration is upward.

we can measure direction of the hand plane. we set up a coordinate system. . If the hand is still.Figure 2: Sign of Measurement of the Accelerometer By placing three accelerometers on the back of the hand. if the finger moves slowly. Moreover. then at each reading its acceleration are mostly due to gravity.

Figure 4 shows the z-axis acceleration output when a hand suddenly moves downward and goes back. Figure 4: Accelerometer Output for moving Hand .Figure 3: Hand Coordinate System If the hand moves very rapidly. the process is a sequence of acceleration-deceleration. Therefore the output will show a large peak and afterwards a peak with opposite sign.

We have to define a system that is ready to go. their outputs are very noisy. In addition. Limited by budget. which poses a difficulty for the user. Hardware/Software Tradeoffs One of the major limitations for our project is the functionalities of the accelerometers. However. Although these accelerometers require simpler hardware and software than a digital or multi-axis accelerometers does. It is difficult to integrate the acceleration to get velocity. This diagram shows the flow of data from when the user end Java Application opens the port for communication to when then information is transmitted to the Application and mapped to a keystroke. but can also be easily expanded. We cannot hard code every possible action. adding more actions meaning adding states in the state machine. Hardware Design .5g. they have a measurement range of only 1. However. we applied several techniques to reduce the noise and were still able to acquire much useful information from the accelerations. available materials and soldering technique. not to mention position. meaning that fast movements will cause the output to reach the rails. Another tradeoff is expandability of the system versus usability. Specifics are covered in Software and Hardware sections.Logical structure Figure 5: Logical Structure The high level design of our project is shown in Figure 5. we only used one-axis low-g analog accelerometers.

the accelerometer circuit. Schematics for all hardware can be found in the Appendix III: Hardware Schematics. . and the Pololu USB AVR programmer. which we used as the serial connection between PC and the custom PCB. ECE 4760 Custom PCB We used the custom PCB designed by ECE 4760 instructor Bruce Land to interface Atmega644 with our circuits.Our hardware consists of three parts: the custom PCB for Atmega644. The board design is shown in Figure 6 and the layout is shown in Figure 7.

three MMA2260D x-axis analog accelerometers. which is desirable.03V. The schematics are shown in Figure 8.Accelerometer The accelerometer circuit consists of four MMA1260D z-axis analog accelerometers. The correspondence of the accelerometers and the pins are shown in the following table: Accelerometer Type Position Pin . Figure 8: ADC Circuit for Accelerometer The accelerometers are connected to Pin A of the MCU for ADC. and the reference voltage is set to Vcc. the circuit decreased the magnitude of the noise in the output by 0. From our tests. and the external circuit. which consists of decoupling capacitors and low pass filtering circuits as the datasheet of MMA1260D and MMA2260D suggests.

5 *Note: Although both accelerometers are X-axis.1 A. The programmer contains two control lines. The lines transmit data 8 bits at a time with no parity and one stop bit.4 A. TX and RX.Z-axis X-axis* Z-axis Z-axis Z-axis X-axis* Thumb Back of the Hand Index Finger Back of the Hand Middle Finger Back of the Hand Table 1: Accelerometers-pins A. it transmits one byte on TX. used to send asynchronous serial communication. so it can be accessed from serial port functions. USB-to-Serial Adaptor We used a Pololu USB AVR programmer as the USB-to-Serial Adaptor.3 A. See picture below for details. It looks like a standard serial port to operating system.0 A. we let them measure different axis by orienting one 90 degrees away from another. When the programmer receives one byte from USB.2 A. Figure 9: Picture and Schematics of the Pololu USB AVR Programmer .

we are able to acquire several pieces of information: hand motion. and the new state depends on the current input and the current state. The figure next summarizes the program structure: Figure 10: Program Structure 1) Analog-to-digital Converter We used the analog-to-digital converters on Atmega644 to obtain acceleration measurements from the 6 accelerometers. which would be in the range [-512. and each finger¶s status. With the acceleration data from the six accelerometers. hand orientation.V_OL=0V. V_OH=5V. The input to the state machine is all these information combined. .Software Design Our program naturally consists of two parts according to the logical structure: hand action recognition codes for the MCU and PC end application. Different aspects of the software are described below in details: Hand Action Recognition The action recognition code makes up most of the code for the MCU. 512]. so we chose V_cc(5V) as the reference voltage. For an accelerometer. We took all 10 bits that VDC read and subtracted by 512 as an acceleration value. We used a state machine design for our program. Each state corresponds to an action.

Figure 11: Acceleration Data of a Finger. After trial and error we set noise threshold = 50 and peak threshold = 150. the ADC input select register. the acceleration data acquired is noisy. whereas oscillations larger than 150 are detected as peaks.e. the ADSC bit is set high. indicating a rapid motion. then the hand is moving to the negative end of z-axis as defined in previous sections. Before beginning a conversion. Oscillations smaller than 50 are regarded as caused by noise and are suppressed. As explained in the hardware tradeoff section. The correspondence between peak sign and motion direction is summarized in table 2: . i. which must be taken care of when we extract information about the hand¶s action. each of the input ports is read in turn by incrementing ADMUX. we decided to set a small noise threshold and a large peak threshold. Together with Noise and Peak Thresholds Hand motion is found by detecting peaks of acceleration. and status of each finger. In order to reduce the effect of noise. For example.During every loop of execution. we wait until the last conversion is done. 2) Raw Data Treatment We need to convert raw acceleration data to three pieces of useful information: hand motion. Figure 11 shows the acceleration data of a moving finger. if the z-axis accelerometer on the back of the hand detects a large negative peak. hand orientation.

For example. Refined gestures are same as basic gestures except that they have more restrictions on input. since a rapid moving finger must produce a peak in acceleration. since we only used one axis accelerometers for the fingers. a finger must be straight if its z-axis acceleration is the same as hand z acceleration. The acceleration with the largest magnitude will indicate which of the axes is in direction with the gravity. refined gestures. Three different statuses are defined for fingers: straight. we were able to retrieve the information we need for the main state machine. Therefore we set the status of an originally bent/straight finger to moving once a peak is detected. and bent otherwise. 3) Action Recognition State Machine The action recognition state machine is the most important and complex component of our whole algorithm. can be tedious and subject to error. If the palm is still and facing upward or downward. hand direction can be directly calculated if the hand is not moving and gravity is dominant. we define that if the previous state is ³Aim´ and the input indicates that the hand moves. and moving. the new direction is the same. and set its status to straight/bent. we used a mixed Mealy and Moore machine design to implement the main logic. We divide hand actions into four groups: basic gestures. Actions are states that depend on the current input and previous state. If hand moves in a different way than the previous direction. bent. Understanding constructing a state machine for more than 30 actions. For example. Detection of finger status depends on hand motion and direction. bent middle finger and ±X downward direction. Based on this classification. and actions. Through these simple methods. the state changes to ³Fire´. we defined gesture ³aim´ to be an input of straight index finger/thumb. . Please note that slow finger motion can only be detected when gravity is a dominant acceleration of both hand and fingers. wait until the reading stabilizes to remove the ³bounce´ in acceleration. we first set up a classification for the actions. and these gestures are assigned meaning.Besides. Basic gestures are the most general actions and only depend on current input. For situations other than this. which corresponds to more than 30 states. dynamic acceleration of the finger is used.

the index of an action state is the index of the previous state plus a multiple of 100. 1) Serial Connection Handling . Control Mapping Manager We chose Java to implement the PC end Control Manager for its support of serial connection and multi platforms. we keep a numbering convention: the index of a refined state is the index of its basic state plus a multiple of 10.Figure 12: Refined Gesture ³Aim´ and Action ³Fire´ To keep this action state library flexible and expandable.

a key configuration file for what gestures the device can detect. and triggers the corresponding key event. we can read the available serial ports for the user to choose. The control manager uses functions provided by this package to handle the serial connection. whereas the PC is the receiver. a key configuration file is shown below: . before the application starts receiving data. and user can choose mapping each action to a keystroke. In our design the MCU is the transmitter.The serial connection is used to transmit the printed strings of hand direction and action obtained by MCU to PC. Screenshots of the interface and a . a long key stroke. or a key combination. These actions are presented in the window. Because the USB-serial converter simulates the USB port as a serial port in the computer. we chose to use an open source package RXTX. Since Suns doesn¶t provide built-in serial connection handling for Java. 2) Key Mapping User Interface The key mapping codes use the Java class Robot to parse the string received from serial port. The application reads predefined strings from a .



Figure 13: Screenshots of Control Mapping Manager Previous Attempts We originally planned to use 3-axis digital accelerometers provided by ST electronics. However. We also tried to use Matlab to implement the PC end control manager. Results of the Design Following are some pictures of the complete hardware setup of our project. but found that Matlab would sometimes crash if an unrecognized string was received. After we switched to Java this problem was solved. implementing the key events in Matlab is significantly harder than in Java. these accelerometers are of package LGA16 and we didn¶t have the time to etch and solder a small PCB for every accelerometer. Besides. .

Speed of execution Overall. transmit the action to PC. jump and fire gestures. and the manager would successfully the left. to acquire accurate reading we cannot measure at a rate more than that. Based on our experience.3Hz. According to datasheet of Atmega644 the time of AD conversion should be significantly less than the response time. so we chose a time of 10ms for each AD conversion. right. so the update frequency. and map the action to an arbitrary key stroke. the results of our device are satisfactory. The total time of updating one gesture is <70 ms. 14. Moreover. should be larger than the bandwidth of typical hand movement (~10Hz). The prototype we implemented can successfully recognize the actions according to predefined states. Since the bandwidth of both MMA 1260 and MMA 2260 is 50 Hz. the electrical response time is 2ms. Our speed of execution is limited by the timing of the A/D conversion and the response time of the accelerometers. we conclude that the controller is sensitive and respond rapidly to hand action. We have tested a mapping file for game Super Mario. .

For each test. hand direction. since it operates on a relative low voltage and didn¶t produce any EM wave.8 Based on our result. the USB port and the power plug. We use insulating tape to completely wrap the metal contacts to make sure no short circuit occurs. and successfully mapped the output to a keyboard event. but as explained before. we make sure the project doesn¶t contain any sharp object or small components. In addition. we conclude that the accuracy was very high for static gestures such as hand gestures.8 Action 45 35 77. finger movement and the overall action. Using the control mapping manager is very easy. and possibly improve our algorithm for noise control by adding a training function. However. the only parts that the user will touch are the device itself. improve PC end manager. while leaving user the freedom to expand the gestures that could be recognized. which are all secure. The accuracy for dynamic movement is low due to noise. design a better hardware layout. and is thus safe for children. we have to state that this project is not suitable for children under age 10 to use. correctly used a state machine to update the action. we made sure the actions were in every directions. our project didn¶t have the issue of interference. We took the approach of predefining all basic movements and most common actions that are suitable for game inputs (such as aim/fire). the device can operate just like simulating keyboard. Conclusions Overall our design met our initial expectation in several aspects.3 Hand Direction 60 60 100. Consequently. . Safety Considerations We enforced safety in several ways. Usability The idea of our project is simple. we wished to search for more suitable accelerometers. and once the manager is set up. It detected hand motion and gesture with a high accuracy. In addition. If we were to do the project once more.Accuracy We have separated tested the accuracy of detecting hand movement. there exists a tradeoff between the expandability of the action library and usability.0 Finger Movement 120 97 80. Test Type Total Tests Correct Accuracy(%) Hand Move 60 47 78.

Standards Our design conforms to the RS232 standard for serial connection. During this process. Appendix I: Source Codes Accelerometer. We understand our limits as inexperienced engineers. Intellectual property considerations As mentioned before. Besides. we always keep design safety in our minds by avoiding using high voltage and making sure the project doesn¶t contain any short circuits. In designing we made efforts to promote understanding of technology. Although the accelerometer technology is mature. we used open source RXTX library to handle serial connection. The wiki page for the library is http://rxtx.php/Main_Page Ethical considerations This project does not pose any ethical considerations. we employ it for a new application and developed a novel and fun control device. and we closely observed the IEEE code of ethics. or nationality. sharp objects or small components. and didn¶t use any design that discriminates toward any particular race. We believe that technology has the purpose of improving people¶s living quality. We didn¶t seek to injure or harm any person or their projects by false and malicious actions. and we have been glad to help fellow classmates whenever they have questions or want to share a resource. all claims we made in this report are honest. There is no conflicting interest for this project. The standard is taken care of by USB-to-serial adaptor on the MCU side and the Java RXTX functions on the PC side. In making our decisions. Neither of us received bribery in any form. we are open to criticisms and suggestions. ECE4760ControlMapper. and will not accept if offered.qbang. and had no intention to hide.c contains all MCU side codes for hand action recognition. . cheat or boast about any observations or conclusions.

00 0.00 Lab 1 1 1 1 MMA1260EG-ND 4 MMA2260EG-ND 3 Lab Lab Appendix IV: Task Division Zhe: Writing Proposal Wrote first-run test program y y . Choose "Run->Run Configurations -> Java Application" Appendix II: Hardware Schematics MMA1260D MMA2260D Custom PC board Appendix III: Cost Details Name Atmega644: Custom PC board: Power Supply White Board Pololu USB Programmer MMA1260: MMA2260 Other Parts Jumper Wires Header Pins Total: $67.contains PC end user interface. Install RXTX library according to instruction contained. 4. Open eclipse and switch workspace to the extracted directory.00 5. 3. 2.00 12 40 1.05 Lab Lab Part# Quantity Unit Price Source 1 8.95 Lab Lab Lab Pololu. To use: 1. Extract the archive.00 6.00 19.

y y y y Soldered Accelerometers Crafted tape wrap and Velcro bindings for wearing accelerometers Tested and calibrated algorithm for detecting hand gesture and motion Designed and implemented user interface Kun: Purchase and Test parts Soldered custom PC board and accelerometers Assembled external circuits Performed background math analysis Tested and wrote action detection algorithm Wrote progress reports and the final report y y y y y y References Data sheets MMA1260: Z-axis accelerometer MMA2260: X-axis accelerometer USB to Serial Adaptor Vendor sites Digikey Pololu Code/designs borrowed from others RXTX: Java Library for serial connection Background sites/paper .

for teaching this course and providing assistance and suggestions Allison Smyth and other 4760 TA¶s for assisting in many aspects of the project Thank you very much! . for their donation of accelerometers. Bruce Land. Prof.ASG: acceleration glove Special Thanks We would like to thank the following individuals and companies: Freescale and ST electronics.