ISSN: 2277-4629 (Online) | ISSN: 2250-1827 (Print)

CPMR-IJT Vol. 2, No. 2, December 2012

Vision Based Real Time Finger Counter for Hand Gesture Recognition
S. Nagarajan* Dr. T. S. Subashini** Dr. V. Ramalingam***

ABSTRACT
Sign Language is a natural way of communication for deaf and mute people around the world. They use different hand gestures to communicate with normal people everywhere. In this paper, a real time finger counter is proposed for counting the number of fingers (1-5) shown in a hand gesture. This system involves five phases namely input video capturing, preprocessing, hand region segmentation, feature extraction and recognition. In the first phase, the gesture video is captured using a low cost USB camera in real time. In the second phase, the image frames of the captured video are resized for further processing. The skin colour of hand region is detected using HSV colour space and morphological operations are performed in the third phase. In the feature extraction phase, the convex hull method is used to detect the boundary points of the segmented binary hand image and the vertices of the convex polygon are determined as the convexity defects i.e. finger tips. Finally, the recognition phase recognizes the gesture number based on the number of convexity defects present in the convex polygon. The experimental results show that this technique gives satisfactory recognition rate.

Keywords: Convex hull, Convexity defects, Feature extraction, HSV colour space, Skin colour segmentation, Sign Language recognition

I. INTRODUCTION
Hand gesture is a kind of non-verbal communication used by the deaf people. They express different types of emotions, orders and even cardinal information by hand gestures only. Sometimes, they can be the only way of communication for the mute persons (sign language) and polices traffic co-ordination in the absence of traffic lights. Every culture follows its own gestures which are not followed universally. Hand gestures provide a separate complimentary modality to speech for expressing ones messages. Hence, a visual interface can be developed for natural interaction between human and computers using hand gestures. The key problem in hand gesture interaction is to make the computers to recognize the hand gestures. The existing approaches for hand gesture recognition use either data gloves or Computer Vision. The former acts as a Sensor for digitizing hand and finger motions into multi-parametric data. The other sensors collect the hand configuration and movement. However, these devices are costlier and it needs skilful training

*, **, *** Department of Computer Science and Engineering, Annamalai University, Tamilnadu, India

www.cpmr.org.in

CPMR-IJT: International Journal of Technology

ISSN: 2277-4629 (Online) | ISSN: 2250-1827 (Print)

CPMR-IJT Vol. 2, No. 2, December 2012

from the users. In contrast, Computer vision based techniques require only a camera for making a natural interaction between humans and computers without use of any special hardware equipments. These systems provide artificial vision rather than biological vision. The bigger challenge in these systems are they need to be background invariant, lighting insensitive, person and camera independent to achieve real time performance. But such systems offer better accuracy and robustness. The enormous applications of hand gesture recognition include Sign language recognition, Visual interface for human computer interaction (HCI), pervasive computing, medical image navigation, remote control of video games. The main objective of the proposed work is to design a vision based interface which aims to recognize the sign language numbers from the gestures by counting the number of fingers based on convex hull approach. The remaining part of this paper is organized as follows. Section II provides details about the related work in this area. Section III presents the key observations and methodology of this work. Section IV submits the experimental results of proposed work. Section V gives the conclusions.

finger tips in [9]. Convex hull algorithm in [10] detects the boundary and centre of co-ordinate of hand image for real time recognition of hand gestures. Recognition of sign language numbers using Thinning of segmented image is proposed in [11]. Image based numbers of Persian Sign Language are recognized using Thinning method in [12]. [13] presents the convex hull algorithm for detecting the boundary points of a 2D object. Various robust and efficient hand segmentation algorithms are elaborated in [14]. Different image morphological operations are referred from [15].

III. PROPOSED METHODOLOGY

The experimental setup of finger counter requires a low cost web cam and preferably a plain background with black or green screen. It does not require any special equipments such as colour data gloves. The system tolerates minor camera motion including vibration. The proposed work is shown in Fig.1 1. The gesture video is captured frame by frame using a webcam as a first step. 2. The captured RGB image is resized into 300 X 200 pixels in the pre-processing stage to improve the processing speed and to reduce the calculation time. 3. Then the image is converted from RGB to HSV colour space [16]. 4. The skin colour [18] of the hand is detected from HSV image by applying a threshold value (less than 18 or greater than 175) for hue component. Here threshold of H is selected as 15. 5. Since the segmented binary hand image contains noise blobs, any small blobs [17] which are less than 100 pixels are removed to filter the image. 6. Convex hull method detects boundary points and centroid of hand. 7. By keeping the largest distance between centre point and the boundary point as radius, the ROI is detected with a bounding box. 8. The vertices of the convex polygon are found out to identify the finger tips or convexity defects.
CPMR-IJT: International Journal of Technology

II. RELATED WORK

The work in [1] presents a new approach for hand gesture analysis for digit recognition. [2] explains a novel vision based method for recognizing finger actions by detecting the fingertips and analysing finger tip parameters. In [3], the authors have proposed a novel algorithm for detecting finger tip extracted using Bayesian rule based skin colour segmentation. The proposed work in [4] uses boundary tracing for identifying number of opened fingers for ASL alphabets. [5] describes the various hand gesture recognition techniques. A new technique in [6] detects the hand, determines its centre, tracks the hands trajectory and analyses the hand locations. The various pixel based skin colour detection techniques for segmenting hand region are compared in [7]. A survey of recent hand gesture recognition systems is presented in [8]. Thinning algorithm is applied to detect
www.cpmr.org.in

ISSN: 2277-4629 (Online) | ISSN: 2250-1827 (Print)

CPMR-IJT Vol. 2, No. 2, December 2012

9. The number of fingers is counted based on the number of convexity defects present in the convex polygon and the count is displayed.

Table 1 Confusion Matrix for of Finger Counter

Number of Fingers counted 2 3 4 5 0 0 0 0 29 0 0 0 2 28 0 0 0 2 28 0 0 0 0 30 % of Recognition 100 96.6 93.3 93.3 100

IV. EXPERIMENTAL RESULTS

The proposed work is implemented in Microsoft Visual Studio 2010 with OpenCV 2.1 and five gestures are tested with 30 different people having different hand size under varying lighting conditions. Each person shows five gestures representing numbers from 1 to 5 in front of the camera. Experiments show that the response time using the Finger Counter is comparable to the time for selecting a key in the keyboard as it does not require a database of training images and it requires virtually no training on the user side, because the gestures which it recognizes are very simple and intuitive. The outputs at different stages of finger counter are given in Fig.2 and the experimental results are tabulated in Table I.
Hand Gesture Capture input image from video Resize the image frame to 300 X 200 Convert the RGB image into HSV colour space Binarise the hand image by thresholding & remove noise Detect boundary points and centroid of hand using convex hull Extract the Region Of Interest Find the Convexity defects (finger tips) Recognize the number of fingers Finger Count

Input Gesture number

1 2 3 4 5

1 30 1 0 0 0

The average rate of recognition of finger counter is 96.64%. The rectangle seen in the output of recognition module represents the bounding box and two blue dots represent the convexity defects which are finger tips. The convexity defects are the outstanding areas of the mask disturbing the continuity of the corresponding convex hull. They are commonly used in gesture recognition in order to identify the potential finger segments. The start and end points of the convex hull wont be counted as defects. Hence the number of fingers counted is given by the formula, Number of Fingers counted=Number of convexity defects-2

Fig.1 Methodology of Finger Counter

www.cpmr.org.in CPMR-IJT: International Journal of Technology

a. Original RGB image b. HSV image

c. Hue component

d. Skin detected

ISSN: 2277-4629 (Online) | ISSN: 2250-1827 (Print)

CPMR-IJT Vol. 2, No. 2, December 2012

International Journal of Computer Science and Information Security (IJCSIS), Vol.2, No.1, 2009. [2] Daeho Lee and SeungGwan Lee, Vision-Based Finger Action Recognition by Angle Detection and Contour Analysis, ETRI Journal, Vol.33, No.3, 2011. [3] M.K. Bhuyan, Debanga Raj Neog and Mithun Kumar Kar, Fingertip detection for Hand pose recognition, International Journal of Computer Science and Engineering (IJCSE), Vol.4, No.3, 2012. g. Convexity defects GUI of finger counter (Finger tips) Fig. 2 Outputs of Finger Counter at different stages [4] Ravikiran J, Kavi Mahesh, Suhas Mahishi, Dheeraj R, Sudheender S, Nitin V Pujari, Finger Detection for Sign Language Recognition, Proceedings of the International Multi Conference of Engineers and Computer Scientists (IMECS), Vol I, 2009. [5] Ginu Thomas, A Review of Various Hand Gesture Recognition Techniques, VSRD International Journal of Electrical, Electronics and Communication Engineering (VSRD-IJEECE), Vol.1(7), pp 374-383, 2011. [6] Mohamed Alsheakhali, Ahmed Skaik, Mohammed Aldahdouh, Mahmoud Alhelou, Hand Gesture Recognition System, The Islamic University of Gaza, Palestine, 2011. [7] Vladimir Vezhnevets, Vassili Sazonov, Alla Andreeva, A Survey on Pixel- Based Skin Colour Detection Techniques, Graphics and Media Laboratory, Mascow State University, Russia. [8] Rafiqul Zaman Khan and Noor Adnan Ibraheem, Hand Gesture Recognition : A Literature Survey, International Journal of Artificial Intelligence & Applications (IJAIA), Vol.3, 4

e. Filtered image

f. Convex hull

V. CONCLUSIONS
In this work, we have focused mainly on hand gesture recognition with a limited set of gestures presenting one up to five fingers of the users hand. The number of defects are used to count fingers ratio but the possibilities are unlimited. In future, this can be extended to check the angle relationship between them, their relative length and so on. By requiring users to make a small number of hand poses and hold their hands parallel to the image plane, the system is able to make reliable recognition in real time. The main drawbacks of this proposed work are hand pose orientation and detection of convexity defects in complex background if any other skin colour object is present near to the users hand. These drawbacks can be overcome in real time hand gesture recognition system with automatic orientation mechanism in future.

VI. REFERENCES
[1] Ahmed BEN JMAA, Walid MAHDI, Yousra BEN JEMAA and Abdelmajid BEN HMADOU, A new approach for digit recognition based on hand gesture analysis,
www.cpmr.org.in

CPMR-IJT: International Journal of Technology

ISSN: 2277-4629 (Online) | ISSN: 2250-1827 (Print)

CPMR-IJT Vol. 2, No. 2, December 2012

[9] Ahmad Yahya Dawod, Junaidi Abdullah, Md. Jahangir Alam, Fingertips detection from colour image with complex background, The 3rd International Conference on Machine Vision (ICMV 2010), 2011. [10] Xianghua Li, Jun-ho An, Jin-hong Min and Kwang-Seok Hong, Hand Gesture Recognition by Stereo Camera using the Thinning Method, IEEE, 2011. [11] Rajeshree Rokade, Dharmpal Doye, Manesh Kokare, Hand Gesture Reccognition by Thinning Method, International Conference on Digital Image Processing, IEEE Computer Society, 2009. [12] Alaa Barkoky, Nasrollah M. Charkari, Persian Sign Language Numbers Recognition using Thinning method, International Journal of Multimedia Technology (IJMT), Vol.2, No.1, pp 27-31, 2012.

[13] Convex Hull, A notes on Introduction to Algorithms, Recitation 24, May 6, 2011. [14] Archana S. Ghotkar & Gajanan K. Kharate, Hand Segmentation Techniques to Hand Gesture Recognition for natural Human Computer Interaction, International Journal of Human Computer Interaction (IJHCI), Vol.3, Issue(1), 2012. [15] R.C.Gonzalaz and R.E.Woods, Digital Image Processing, 2nd ed., Pearson education, 2002. [16] http://en.wikipedia.org/wiki/HSL_and_HSV [17] http://www.shervinemami.info/blobs.html [18] Ciarn Conaire, Noel E. OConnor and Alan F. Smeaton, Detector adaptation by maximising agreement between independent data sources, IEEE International Workshop on Object Tracking and Classification Beyond the Visible Spectrum, 2007.

www.cpmr.org.in

CPMR-IJT: International Journal of Technology