Device Agnostic 3D Gesture Recognition using Hidden Markov Models

Anthony Whitehead, Kaitlyn Fox

School of Information Technology
Carleton University
1125 Colonel By Drive, Ottawa, Ontario, Canada
{awhitehe},{kfox2}@connect.carleton.ca

ABSTRACT
Hidden Markov Models have been effectively used in pattern
recognition systems in the past. In this work, we identify the ne-
cessary elements to successfully use an HMM system for 3D ges-
ture recognition regardless of the sensor device being used. So
long as the sensor system itself is capable of outputting informa-
tion about the 3 axes of motion (X, Y, and Z), that information
can be used in this generic model for accurate, high speed gesture
recognition. The proposed system works with accelerometer data,
positional data and gyro data alike.
Categories and Subject Descriptors
H.5.2 [II nfor mation I nter faces and Pr esentation]: User Interfaces-Input
devices and strategies; K.8.0 [P Per sonal Computing]: General-Games;
Patter n Recognition]: Implementation - Interactive systems
I.5.5 [P
General Terms
Performance, Design, Experimentation , Human Factors, Theory.
Keywords
Gestures, HMM, Games.
1. INTRODUCTION
A hidden Markov model (HMM) is a statistical model, initially
developed for speech recognition [1]. Some current applications
of HMMs in gesture recognition include, among others, work by
Kratz[2], Kreskin[3], and Segen[4]. In a hidden Markov model, a
sequence is modeled as an output generated by a stochastic
process progressing through discrete time steps. At each time step,
the process outputs a symbol (from a predefined alphabet) and
moves from one state to the next state. Both actions, the transition
from state to state and the emission of an alphabet symbol, follow
probability distributions that define the model. Thus we can esti-
mate these probabilities using a training by example process.
In a hidden Markov model, only the sequence of emitted symbols
is observed. The path of states followed by the process is “hid-
den” from the observer. Given a hidden Markov model (trained by
examples), M, and an input sequence S (a gesture), the standard
question is whether S has the properties of the model M. Less
precisely: Does the input gesture “look like” the training gestures?
To answer, we compute the probability, P[S|M], of an observed
sequence S (an input gesture) being generated by the (trained)
model M. The log of the ratio of P[S|M] to the probability of ge-
nerating S by chance is usually used an assessment function.
2. Decomposing the HMM
In order to remain device agnostic, the goal is to ensure that a
mapping of sensor inputs to a common alphabet of inputs is de-
manded. This may seem quite difficult at first glance given that
different sensors will be recording different types of information
that are not altogether related. However, given the discrete nature
of any sensor, it is relatively simple to have symbols output based
on the discrete segmentation of the sensors range. Suffice it to
say, that granularity of the decomposition depends primarily on
the precision of the sensors being used. However, it does not
invalidate the model; rather it simply expands the alphabet. The
only issue that remains is the computational requirements for an
increased alphabet. It is also important to note that this decompo-
sition requires a 1:1 alphabet to state. i.e. each state outputs a
unique symbol. Since our gestures are in 3D space, and our sen-
sors are emitting 3-dimensional data, we carve our space into
discrete sub-cubes. In our tests, we have experimented with dis-
crete segmentations of 3,4 and 5 subspaces in each dimension
resulting in alphabet sizes of 27, 64 and 125. Our experiments
show that alphabet sizes of 27 allow real-time interactivity; the
larger sizes are still too computationally expensive for effective
real time applications.
Figure 1. 7 Gestures

Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
FuturePlay @ GDC Canada, May 12-13 2009, Vancouver, Canada.
Copyright 2009
2.1 Gestures in Training
To record gesture data, a sensor was placed on a subject’s wrist.
We gathered and stored sensor data for the duration of a gesture
performance. This procedure was repeated multiple times with
several subjects to generate the training set. Gestures used in our
experiments are shown below.

29
2.2 The Number of HMM States
A balance needs to be struck between false negative results and
false positive results in order for the system to be usable. We
experimented with 27, 64, and 125 states in the HMM. Overall,
the best results were achieved using 27 states, with the exception
of the false positive rate which increases slightly at 27 states. This
slight increase is negligible however due to the significant in-
crease in correct classification rate attained at 27 states. This ex-
periment also illustrates that more training samples provided will
result in better recognition rates.
Figure 2: Correct Classification Rates for 27, 64 and 125 state
HMMs
The side benefit of using a smaller number of states is that the
recognition time decreases significantly. We are able to achieve
800 gesture recognition tests per second on a 27 state HMM (2.6
GHz processor) while the 125 state HMM runs a single recogni-
tion task in the order of seconds.
2.3 Number of Samples in a Training Set
Previous experiments have shown that several subjects’ data is
required for easy replication by the general population. We con-
ducted several experiments to determine the optimal number of
samples required to generalize the gesture training data.
Figure 3. Effect of Training Sample Size on Classification
Rates for 27, 64 and 125 state HMMs
As the number of gesture samples increases, the curves of the
graphs begin to flatten. Based on these results, a set of around 250
samples in a training set is a sufficient and feasible amount of data
to collect while still providing good detection results. It should be
noted that the more states the HMM has, the more training sam-
ples are needed to achieve higher classification rates. In our expe-
riments, since we have a real time interaction constraint, we are
effectively limited to 27 and 64 states as 125 does not compute
results quickly enough.
2.4 Culling Training Data
We also found that during the data collection phase there were a
number of issues with the samples being inconsistent. For exam-
ple the training set was created by pressing a button on the device,
performing the gesture and letting go of the button. In this case,
delays in button presses and releases inaccurately skew the statis-
tics. This is solved by culling the input data based on sequence
lengths. We throw away the longest and the shortest sequences.
We used a 1.5 standard deviation rule to decide which sequences
were to be removed.
2.5 Grouping Data to Lower False Positives
We selected our gestures as part of a game design, rather than
being arbitrarily chosen. This created a set of gestures that had
significantly different lengths and allowed us to further reduce
false positive rates by using a linear discriminator to first classify
the gestures into short gestures and long gestures. Any gesture
sequences with fewer than 27 elements were considered short and
any with more than 27 were considered long.
During our live testing, no long moves performed were ever
shorter than 28 elements and only one short move (out of 70 tri-
als) was longer than 27 elements, thus effectively removing the
false positives that occurred between small and large moves.
2.6 Left Hand vs. Right Hand Training
We conducted an experiment to determine whether gestures
trained by the right hand could be detected if the sensor was
placed on the left hand for testing. As expected, for the majority
of gestures, the left hand tests were not recognized as valid ges-
tures when compared to the right hand training data. This is be-
cause our left and right arms have a different “natural tilt” to them
which result in different sensor values when the mirrored gestures
are performed. Rates dropped by more than half in most cases.
However, some gestures that are unidirectional and involve no
roll or tilt, can be performed fairly easily by both hands. For op-
timal results in gesture recognition, it is recommended that each
gesture for each limb should be trained individually.
3. RESULTS AND CONCLUSION
We have achieved very good recognition rates (in the table below)
using the above stated decomposition of the HMM model which
is applicable in a sensor agnostic way.
F +ve T +ve F -ve T -ve Correct
In training set 11.2% 94.4% 5.6% 88.8% 91.6%
Not in training 11.4% 84.1% 15.9% 88.6% 86.4%
set
Table 1: Recognition results for tests for all 7 gestures when
the person is part of the training set and not.
The decomposition technique allows for any 3D sensing device to
use a discrete HMM as part of a gesture recognition system while
the speed allows for multiple sensors to be used together.
4. REFERENCES
[1] Rabiner, L. A tutorial on hidden Markov models and se-
lected applications in speech recognition, Readings in speech
recognition, Morgan Kaufmann Inc., San Fran., CA, 1990
[2] Kratz, L., Smith, M., and Lee, F. J. Wiizards: 3D gesture
recognition for game play input. In Proceedings of the 2007
Conference on Future Play Toronto, Canada, Nov '07.
[3] C. Keskin, A. Erkan, and L. Akarun. Real time hand tracking
and 3d gesture recognition for interactive interfaces using
hmm. In Proceedings of the Joint International Conference
ICANN/ICONIP 2003. Springer.
[4] Segen, J. and Kumar, S. Fast and Accurate 3D Gesture Rec-
ognition Interface. In Proc of the 14th ICPR. IEEE Comput-
er Society, Washington, DC, August 1998.
30