Android :Single Stroke Gesture Recognition using $1 Detector

Pi19404
April 5, 2013

Contents

Contents
Android :Single Stroke Gesture Recognition using $1 Detector
0.1 Introduction . . . . . . . . . . . . . . . . . . . . . 0.2 $1 Unistroke Recognizer . . . . . . . . . . . . . 0.3 Gesture Normalization . . . . . . . . . . . . . . . 0.3.1 Registering candidate . . . . . . . . . . 0.3.2 Rejecting invalid Gesture . . . . . . . . 0.3.3 Re-sampling points . . . . . . . . . . . . . 0.3.4 Scaling . . . . . . . . . . . . . . . . . . . . . 0.3.5 Translation . . . . . . . . . . . . . . . . . . 0.4 Rotational Invariance . . . . . . . . . . . . . . . . 0.5 Rejecting gestures . . . . . . . . . . . . . . . . . 0.6 Robustness to rotation . . . . . . . . . . . . . . 0.7 Computing the Similarity Score . . . . . . . . . 0.7.1 Loading the Template Gestures . . . 0.7.2 Android Gesture Capture Application 0.8 Launch Application . . . . . . . . . . . . . . . . . 0.9 Code . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3
3 3 4 4 5 5 7 7 8 8 9 9 10 12 12 12 12

2 | 13

Android :Single Stroke Gesture Recognition using $1 Detector

Android :Single Stroke Gesture Recognition using $1 Detector

0.1 Introduction
In this article will look at single stroke gesture recognition on android platform using $1 Unistroke Recognizer by Anthony and Wobbrock, 2010

0.2 $1 Unistroke Recognizer

$1 Unistroke Recognizer is a method to recognize single/uni stroke gestures made by the user using the mobile touch interface.The method is simple,computationally efficient and can recognize a wide variety of gesture and is suitable for mobile based platforms. A userAZs gesture results in a set of candidate points g , and we must determine which set of previously recorded template points i it most closely matches. The candidate and template points are obtained by interactive mean and in the present application the candidate points are obtained by using the mobile touch screen interface.The candidate points are spatially sampled at the rate determined by the touch screen interface on the mobile device. The template are constructed using desktop applications using mouse to draw gesture and hence are spatially sampled at the rate determined by the desktop-mouse interface. In the present implementation we will assume that the raw template points have been captured using a desktop application and sampled points are stored in a file. In the present application we will ignore the speed at which the

3 | 13

Android :Single Stroke Gesture Recognition using $1 Detector gestures are drawn.

0.3 Gesture Normalization

The template and the candidate points may contain different number of sampled points,they may differ in size and spatial location are not the same ie the template and candidate points would not line up. Hence the first step is to perform gesture normalization so that candidate and template points are transformed such that they can be compared this pre processing step is called as gesture normalization. The aim to transform the gesture so that they are invariant to translation and rotation

0.3.1 Registering candidate

The first step is to capture the candidate template.This step is called registering the candidate. As mentioned earlier the number of points captured would depend on the device spatial resolution. The gesture capture processing is defined to be in one of the three states dragged,released,start. The start state indicates that gesture has started and to clear any previous information stored. The dragged state indicates that unistroke is being performed without lifting the figure and the 2D co-ordinates of gesture are being captured.The released state indicates that figure has been lifted and gesture capture process has been completed and to start with gesture recognition process. The class AndroidDollar defines android routines to capture the touch gesture performed by the user . The class Data Capture provide high level interface to capture the data and to initiate gesture recognition.AndroidDollar class contains a instance of DataCapture and whose methods are called based on the touch event detected by the user The java class DataVector is defined which captures the 2D co-

4 | 13

Android :Single Stroke Gesture Recognition using $1 Detector ordinate information of drawn gesture. contains instance of DataVector. The DataCapture Class

The class PUtils contains all the methods for gesture pre processing and recognition. The DataCapture class contains instance of PUtils class .

0.3.2 Rejecting invalid Gesture

A simple check is incorporated to check whether the gesture was intentional or not by specifying a path length criteria.If the path length of the gesture is less than a specified threshold no further processing is performed and no gesture recognized status will be displayed. The PathLength method defined in the PUtils class simply computes the sum of linear distances between all the adjacent points of the captured/template gesture.

1 2 3 4 5 6 7 8 9

public double PathLength ( Vector points ) { double length = 0; for ( int i = 1; i < points . size () ; i ++) { length += Distance (( Point ) points . elementAt ( i - 1) , ( Point ) points . elementAt ( i ) ) ; } return length ; }

In the present implementation the path length threshold used in 100. The code is implemented by the method PathLength is the PUtils Class.

0.3.3 Re-sampling points

Once the gesture has been captured before the process of comparing the candidate gesture with template gesture some pre processing operations are performed .Resampling is one such operation.

5 | 13

Android :Single Stroke Gesture Recognition using $1 Detector The re-sampling operations selects from the provided candidate/template gesture a fixed subset of points. This ensures than candidate and template have the same number of points enabling us to perform point based comparison. The method used for sampling the data points is uniform sampling.The path length is divided by the number of re-sampled points.This will be the interval length between the points. We start with the initial point and next point is selected such that distance between points is greater than of equal to interval length.Let points be labeled pt1 and pt2 A linear path is assume to exist between adjacent sample points. using simple trigonometric relationship of sin and cos we can estimate of location at distance of uniform path interval which lies between pt1 and pt2 . This new co-ordinate replaces pt2 in the candidate/template coordinate array and the same process is repeated till the last point of the co-ordinate array is reached.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

double d = Distance ( pt1 , pt2 ) ; if (( D + d ) >= I ) { // computation of new co - ordinate using cos relationship double qx = pt1 . x + ( I / ( D + d ) ) * ( pt2 . x - pt1 . x ) ; // computation of new co - ordinate using sin relationship double qy = pt1 . y + ( I / ( D + d ) ) * ( pt2 . y - pt1 . y ) ; Point q = new Point ( qx , qy ) ; // adding the point in resampled array dstPts . addElement ( q ) ; // replacing the point in the source array srcPts . insertElementAt (q , i ) ; // resetting cumulative distance D = 0.0; } else { // computing cumulative distance D=D+d; }

This is implemented by the method Re-sample in the PUtils Class. In the present implementation the number of re-sampled points

6 | 13

Android :Single Stroke Gesture Recognition using $1 Detector used is 32.

0.3.4 Scaling
The next pre-processing step is to scale the co-ordinates such that third width and height remain within a fixed bounds.First the bounding width and height of current set of points are computed which is simply max(x) min(x) and max(y ) min(y ).The new width and height are denoted by W and H and all the points are scaled by factor W=max(x) min(x) and H=max(y ) min(y )
1 2 3 4 5 6 7 8 9

Rectangle B = BoundingBox ( points ) ; Vector newpoints = new Vector ( points . size () ) ; for ( int i = 0; i < points . size () ; i ++) { Point p = ( Point ) points . elementAt ( i ) ; double qx = p . x * ( size / B . Width ) ; double qy = p . y * ( size1 / B . Height ) ; newpoints . addElement ( new Point ( qx , qy ) ) ; }

This provides step provides invariance wrt scaling since all gestures are bounded to lie within rectangle of same size.This is implements by method ScaleToSquare1 in PUtils class. In the present implementation the scale is done so than bounding box is a square of dimension 250. The above method perform uniform scaling another method of scaling is non uniform scaling that maintains the aspect ratio. Compute the ration between the width and height or viceversa of the bounding rectangle if ratio is close to 0 than 1 then perform non uniform scaling else perform uniform scaling. The ScaleDimTo method implements this in the PUtils Class.

0.3.5 Translation
The first step required is computation of mean/centroid of set of co-ordinate location.
1 2 3 4

Enumeration e = points . elements () ; while ( e . hasMoreElements () ) { Point p = ( Point ) e . nextElement () ;

7 | 13

Android :Single Stroke Gesture Recognition using $1 Detector

5 6 7 8

} return new Point ( xsum / points . size () , ysum / points . size () ) ;

xsum += p . x ; ysum += p . y ;

This is implemented by the method Centroid in class PUtils. The next step is to translate all the points such that centroid lies at the origin of co-ordinate system. Translate all the points by (centroidx ; centroidy ).

1 2 3 4 5 6 7 8 9

Point c = Centroid ( points ) ; Vector newpoints = new Vector ( points . size () ) ; for ( int i = 0; i < points . size () ; i ++) { Point p = ( Point ) points . elementAt ( i ) ; double qx = p . x - c . x ; double qy = p . y - c . y ; newpoints . addElement ( new Point ( qx , qy ) ) ; }

This is implemented in the method TranslateToOrigin in the PUtils Class.

0.4 Rotational Invariance

If rotational invariance is desired.The line joining the first point to the origin can be rotated so that it lies along 0 degree axis.This will ensure that all template are aligned . The RotateToZero method in PUtils class performs this action.

0.5 Rejecting gestures

After the pre processing step the candidate and template gesture are ready to be compared. However we can reject templates by incorporating crude comparison. compute the normalized vector 1 between the starting point and the 8 of gesture length. This is done by computing difference in x and y co-ordinate of the

8 | 13

Android :Single Stroke Gesture Recognition using $1 Detector two points normalized by the distance between them.This vector is called StartUnitVector

1 2 3 4 5

Point i1 =( Point ) points . elementAt ( index ) ; Point i2 =( Point ) points . elementAt (0) ; Point v = new Point ( i1 . x - i2 .x , i1 . y - i2 . y ) ; double len = Math . sqrt ( v . x * v . x + v . y * v . y ) ; return new Point ( v . x / len , v . y / len ) ;

CalcStartUnitVector method of PUtils class computes this angle. To compare startunitvectors of template and candidate cosine of angle between the unit vectors is used as criteria The AngleBetweenUnitVectors method compute difference in angle
1 2

double n = ( v1 . x * v2 . x + v1 . y * v2 . y ) ; return Math . acos ( n ) ; // arc cosine of the vector dot product

In the present implementation the angle similarity threshold is set to 30 deg.

0.6 Robustness to rotation

Although above pre-processing steps normalize the gesture wrt scale and translation,every time a gesture is drawn there may be subtle changes like small change in orientation.Hence some robustness to rotation is required to provide a good score. This is performed by rotating the candidate points about line joining the starting point and the centroid by  and + in small increments and computing score at each incremental angle. The best matching score amongst all the angles is selected.

0.7 Computing the Similarity Score

The final step is to compute the similarity score

9 | 13

Android :Single Stroke Gesture Recognition using $1 Detector As method earlier a scan of angle withing a specified range is performed. It is assumed that maximum score exits amongst the angle. Instead of performing a uniform search the Golden section search algorithm is used to search for global maximum from list of candidate by by successively narrowing the range of values inside which the maximum score is known to exist. The cost function or score is provided DistanceAtAngle function which computes the distance between the candidate rotated by specified angle and template.The score is compute by computing the euclidean distance between the 2D set of vectors. DistanceAtBestAngle method in PUtils Class compute the best score by applying golden section scan algorithm about range of specified angles. DistanceAtAngle method in PUtils Class performs transformation about the points such that all points are rotated by specified angle before comparison. PathDistance method in PUtils class computes the euclidean distance between the set of 2D points.

1 2 3 4 5 6

double distance = 0; for ( int i = 0; i < path1 . size () ; i ++) { distance += Distance (( Point ) path1 . elementAt ( i ) , ( Point ) path2 . elementAt ( i ) ) ; } return distance / path1 . size () ;

The Recognize method in PUtils class provides interface to gesture recognition routine.

0.7.1 Loading the Template Gestures

The template gesture were captured using desktop application and co-ordinates are stored in a csv files .A directory is create for each class of template and multiple files can be placed withing each

10 | 13

Android :Single Stroke Gesture Recognition using $1 Detector directory.For each files a template object will be created. In the CSV files raw x and y co-ordinates of gesture trajectory of mouse pointer captured by desktop application are written side by side and this entire CSV string is loaded into a integer array.Hence while loading the template nth location is assigned to x co-ordinate while (n + 1)th location of integer array is assigned to y co-ordinate of 2D co-ordinate object representing Point Class. The templates are loaded once during initialization. Comparison is made with all the loaded templates and best score is selected amongst the templates. Below are examples of templates demonstrated in the present application.The template consist of following shapes made in clockwise and anti clockwise manner.The templates are labeled as (1,11) ,(2,12) ,(3,15) (9,13),(10,14) for clockwise and anticlockwise respectively. An example of data contained in CSV files is shown below

75,154,81,150,85,146,90,142,96,139,101,135,105,130,110,128,114,124, 119,120,122,117,125,113,128,108,130,103,132,96,133,93,133,89,133,85, 133,81,133,75,133,72,133,69,133,65,131,62,128,60,125,59,122,57,119,57, 116,57,113,56,110,56,108,56,107,57,105,62,104,67,103,71,103,75,103,78,103, 81,103,86,103,90,103,93,103,97,104,101,105,104,107,107,109,111,111,116, 113,119,115,123,119,127,122,130,126,133,129,136,133,139,136,

11 | 13

Android :Single Stroke Gesture Recognition using $1 Detector 142,140,147,143,150,147,153,150,155,152,157,155,158 This contains x and y co-ordinates stored in adjacent locations.

0.7.2 Android Gesture Capture Application

The android gesture capture exampled code is based on the TouchPaint example provided in the android samples directory under graphics sub-directory.Few modifications we made to gesture capture interface. The AndroidGesture capture utility will contain a instance of DataCapture which provides high level interface to gesture registering and recognizing methods. When the gesture is desired to be captured On method of DataCapture class is called with the first boolean parameter set to true and second,third parameters are x and y co-ordinates respectively. When the gesture is completed the On method of DataCapture Class is called with first boolean parameter set to false and second and third parameters are the final x and y co-ordinates respectively.

0.8 Launch Application

Transfer the apk file generated to device and test the application.

0.9 Code
The code can be found in code repository https://github.com/ pi19404/m19404/tree/master/Android/AndroidGesture or https://code. google.com/p/m19404/source/browse/Android/AndroidGesture. The header files is located in jni directory.The library files are not placed in the repository download them from appropriate packages on send a mail separately for download link.

12 | 13

Bibliography

Bibliography
[1] Lisa Anthony and Jacob O. Wobbrock.  A lightweight multistroke recognizer for user interface prototypes. In: Proceedings of Graphics Interface 2010. GI '10. Ottawa, Ontario, Canada: Canadian Information Processing Society, 2010, pp. 245 252.

1839214.1839258.
[2]

isbn: 978-1-56881-712-5. url: http://dl.acm.org/citation.cfm?id=

Gary R. Bradski.  Computer Vision Face Tracking For Use in a Perceptual User

url: http://citeseerx. ist.psu.edu/viewdoc/summary?doi=10.1.1.14.7673.

Interface. In: Intel Technology Journal Q2 (1998).

[3]

Paul A. Viola and Michael J. Jones.  Rapid Object Detection using a Boosted Cascade of Simple Features. In: CVPR (1). 2001, pp. 511518.

13 | 13