Online Hand Gesture Recognition &
Classification for Deaf & Dumb
Nitesh S. Soni1, Prof. Dr. M. S. Nagmode2, Mr. R. D. Komati3
Student1, Professor2, Ass. Professor3
Department of Electronics and Telecommunication
MIT College of Engineering, Pune
{soninitesh.nitesh1, manoj.nagmode2, rdkomati3}@gmail.com
ABSTRACT: Sign language is widely used by
individuals with hearing impairment to communicate
with each other conveniently using hand gestures.
However, non-sign-language speakers find it very
difficult to communicate with those with speech or
hearing impairment since it interpreters are not
readily available at all times. Many countries have
their own sign language, such as American Sign
Language (ASL) which is mainly used in the United
States and the English-speaking part of Canada. The
proposed system helps non-sign-language speakers in
recognising gestures used in American Sign
Language. The system described in this paper is
implemented using MATLAB. In this approach,
firstly, the signs are captured using a webcam. The
images captured are then processed further and the
features are extracted from the captured images using
PCA. Comparison of the features is done using
Euclidean Distance with the training sets. Minimum
Euclidean distance helps to recognise the character.
This system will enable non-sign-language speakers
to better understand and communicate with those
with impaired hearing.
KEYWORDS: Sign language, Euclidean Distance,
PCA.
1. INTRODUCTION
The use of sign language is not only limited to
individuals with impaired hearing or speech to
communicate with each other or non-sign-language
speakers and it is often considered as a prominent
medium of communication. Instead of acoustically
conveyed sound patterns, sign language uses manual
communication to convey meaning. It combines hand
gestures, facial expressions along with movements of
other body parts such as eyes, legs, etc. This paper
proposes a design for recognizing signs used in ASL
and interpreting them. Some of the challenges
experienced by speech and hard of hearing people
while communicating with normal people were social
interaction, communication disparity, education,
behavioural problems, mental health, and safety
concerns [3]. The ways in which one can interact
with computer are either by using devices like
keyboard, mouse or via audio signals, while the
former always needs a physical contact and the latter
is prone to noise and disturbances [3].
Physical action carried by the hand, eye, or any
part of the body can be considered as gesture. Hand
gestures are the most suitable and easily interpretable
for humans. Here, single handed gesture recognition
system is proposed, it uses right handed gestures, and
are classified and recognised for the specific
character. Static gesture recognition system proposed
here does not require any colour code. The sign
recognition system proposed, recognizes the signs
with great accuracy and with less overheads in
features and time.
2. LITERATURE SURVEY
Many novel methods have been developed in
past few years, to facilitate communication between
the sign language users and those who can’t speak
sign language. Mrs.Neela Harish, Dr.S.Poonguzhali
[1], proposed a system which depends on flex
sensors, accelerometers output values such as, co-
ordinates given by accelerometer and the bending
values given by the flex sensors, for the interpretation
of signs. Depending on the. Suganya R,
Dr.T.Meeradevi, [2] implemented a system using
Feed Forward Neural Network, for identifying hand
gestures characteristics & to train gestures captured.
Dhananjai Bajpai, et al. [4], proposed a system
enabled to decode more than 36 gestures using 6 flex
sensors, to recognise corresponding alphabets, or
corresponding features. Pre-requisite for using this
system to perform gesture recognition and store it in
the database is not needed. Al-Ahdal and Tahir [7]
proposed a method using data glove and EMG
Sensors for designing SLR. Hand muscles
electromyography signals are recorded, for allocating
words from streams of characters in continuous SLR.
Vision based systems mainly uses colour information
for detection of hand. They use skin colour models or
different colour gloves [10].
3. BLOCK SCHEMATIC
Input Pre- Feature
Image Processi Extractio
Sign Training
Recogniti- Classificat- & 3.3 Feature Extraction
on ion
Testing
3.1 Input image
Input image through web cam is captured and it can
be used to store as dataset for training or as input
image to recognise the character. The image is
captured and stored in any supported format specified
by the device. The resolution of the camera and the
supported format of the connected device, i.e. web
cam can be found using the command:
“imaqhwinfo('winvideo')”
This command provides the details such as:
AdaptorDllName: [1x81 char]
AdaptorDllVersion: '4.5 (R2013a)'
AdaptorName: 'winvideo'
DeviceIDs: {[1]}
DeviceInfo: [1x1 struct]
Using device ID we can make get the supported
format for the images to be captured and stored by
the web cam, as we have used web cam for training
and testing of the proposed system
3.2 Pre-Processing
Pre-processing is required on every image to enhance
the functionality of image processing. Captured
images are in the RGB format. The pixel values and
the dimensionality of the captured images is very
high. As images are matrices and mathematical
operations are performed on images are the
mathematical operations on matrices. So we convert
the RGB image into Gray image using “rgb2gray”
function and thus converting the Gray image into
Binary image. Image segmentation technique is used
on the Binary image to detect the hand region.
Segmentation is done to divide image into two
regions, background and the foreground containing
region of interest i.e. hand region. The segmented
image has the hand region with the pixel value ‘1’
and the background as the ‘0’. The image is then used
as a mask to get the hand region from the RGB image
by multiplying the black and white image i.e. binary
image with the original RGB image. The image is
resized to reduce size of the matrix used for the
recognition process. The images are then converted
into column matrix for feature extraction.
Feature extraction is the most significant step in
recognition stage as the size of the data
dimensionality is reduced in this stage. Here, features
are extracted using Principle Component Analysis
(PCA), where we can get Eigen Values and Eigen
Vectors. To calculate this Eigen Values and Eigen
Vectors, column matrix of all the images is formed
and concatenated to form single matrix, mean of this
matrix is calculated and it is subtracted for
normalisation. The mean of the matrix is calculated
as:
μ= ∑ n
where, M is the number of column matrix.
Mean from each of the column vector of the database
Ti is subtracted as:.
Temp = A- μ
where, Ais column matrix & μ is the mean.
3.4 Training and testing phase
Detection of human hand in white background will
enhance the performance of image pre-processing, in
terms of accuracy and speed. Training phase and the
recognition phase are two Sign recognition phases.
Training phase: Training phase is the first phase,
where the generation and storing of dataset is done.
Number of images per character stored is directly
proportional to the accuracy of the system. During
generation of the training set the images are pre-
processed. Column matrix is generated for each of
the images of dataset and concatenated matrix is
formed. Using this matrix Eigen vector is calculated
which is used to get feature vector. This feature
vectors are used further for classification process.
Testing phase: This stage is the second stage which
requires the input gesture, which is to be recognised.
Input image is normalised with the mean calculated
of the dataset, and Eigen vectors are used for
projecting the input image on the dataset. Maximum
score using Euclidean Distance is calculated and the
gesture is recognised to display the recognised
character to corresponding input gesture.
11 … 1
A= ⋮ ⋮ ⋮
1 ⋯
Where, A is the dataset column matrix, m = Size of
image & n = Number of Images in dataset.
Euclidean Distance is calculated as:
which is used to get feature vector. This feature
vectors are used further for classification process.
Testing phase: This stage is the second stage which
requires the input gesture, which is to be recognised.
Input image is normalised with the mean calculated
of the dataset, and Eigen vectors are used for
projecting the input image on the dataset. Maximum
score using Euclidean Distance is calculated and the
gesture is recognised to display the recognised
character to corresponding input gesture.
4. Dataset
Single image of each character from
dataset for gesture recognition is shown. Increasing

= , − ,

Where, j =1,2,…,m, n=total pixels in single image

m= total number of images in the dataset

3.5 Classification

Euclidean Distance is used to classify the gestures of

hand. Feature vectors calculated for the dataset and
the Feature vector calculated for the input image are
used to classify the character related to the input
gesture. Classification of character is based on the
number of images in dataset will increase accuracy of
maximum score. Maximum score is the maximum
matching score i.e. minimum distance of input Fig.2 Dataset of few characters
gesture with the dataset image of the corresponding the recognition. For creation of dataset we used pre-
character. processing, PCA algorithm on the captured image
and thus created image is resized and saved.
3.6 Sign Recognition
5. RESULTS
Detection of human hand in white background will
enhance the performance of image pre-processing, in The input gesture is taken online as well as
terms of accuracy and speed. Training phase and the offline. The taken image is pre-processed and resized.
recognition phase are two Sign recognition phases. Principle Component Analysis is done to get the
Training phase: Training phase is the first phase, Eigen Vector and Euclidean Distance is calculated
where the generation and storing of dataset is done. between the training set and the input gesture. The
Number of images per character stored is directly accuracy table shows the percentage accuracy for
proportional to the accuracy of the system. During various character.
generation of the training set the images are pre-
processed. Column matrix is generated for each of
the images of dataset and concatenated matrix is
formed. Using this matrix Eigen vector is calculated
 directly with normal people without any knowledge
of Sign Language.
The accuracy of the system depends on the
illumination conditions. Under good illumination and
plain light coloured background we achieved nearly
98% accuracy.

REFERENCES
[1] Mrs. NeelaHarish , Dr. S .Poonguzhali, “Design and
development of hand gesture recognition system for speech
impaired people”, International Conference on Industrial
Instrumentation and Control (ICIC) College of Engineering
Pune, India. May 28-30, 2015.
[2] Suganya R, Dr.T.Meeradevi, “Design Of A Communication
Aid For Physically Challenged”,in IEEE Sponsored 2nd
International Conference On Electronics And
Fig.3 Results of character Y Communication System (Icecs 2015),2015.
[3] P Raghu Veera Chowdary, M Nagendra Babu, Thadigotla
Character Offline mode Online mode Venkata Subbareddy, Bommepalli Madhava Reddy, V
Elamaran, “Image Processing Algorithms for Gesture
Recognition using MATLAB”, IEEE International
Conference on Advanced Communication Control and
A 96% 93% Computing Technologies (ICACCCT), 2014.
[4] Meenakshi Panwar and Pawan Singh Mehra, “Hand Gesture
Recognition for Human Computer Interaction”,
International Conference on Computing, Communication
C 92% 96% and Applications (ICCCA), Dindigul, Tamilnadu, India,
Feburary 2012.
[5] Liton Chandra Paul, Abdulla Al Sumam, “Face Recognition
D 93% 97% Using Principal Component Analysis Method”, in
International Journal of Advanced Research in Computer
Engineering & Technology (IJARCET) Volume 1, Issue 9,
November 2012.
G 97% 97% [6] Adi Izhar Che Ani, Anis Diyana Rosli, Rohaiza Baharudin,
Mohd Hussaini Abbas, Mohd Firdaus Abdullah,
“Preliminary Study of Recognizing Alphabet Letter via
L 96% 94% Hand Gesture”, in Computational Science and Technology
(ICCST), 2014 International Conference, Kota Kinabalu,
India, August 2014.
[7] M. Ebrahim AI-Ahdal & Nooritawati Md Tahir," Review in
W 97% 94% Sign Language Recognition Systems" Symposium on
Computer & InfOlmatics(ISCI),pp:52-57, IEEE ,2012
Sruthi Upendran, Thamizharasi. A, “American Sign
Y 97% 96% Language Interpreter System for Deaf and Dumb
Individuals” in International Conference on Control,
Instrumentation, Communication and Computational
Table1. Accuracy Table Technologies (ICCICCT), 2014.
[8] A. Sharmila Konwar, B. Sagarika Borah, C. Dr.T.Tuithung,
“An American Sign Language Detection System using HSV
6. CONCLUSION Color Model and Edge Detection”, International Conference
on Communication and Signal Processing, April 3-5, 2014.
Hard of hearing and mute people make use of sign [9] Pravin R Futane , Rajiv v Dharaskar, “Hasta Mudra
language to communicate. Signs, such as hand anvinterpretatoin of Indian sign hand gestures”,
international conference on digital object identifier, vol.2,
gestures are used in many sign languages, for the pp:377-380, IEEE ,2011.
purpose of this project, American Sign Language has [10] Adi Izhar Che Ani, Anis Diyana Rosli, Rohaiza Baharudin,
Mohd Hussaini Abbas, Mohd Firdaus Abdullah,
been used. Principle Component Analysis technique “Preliminary Study of Recognizing Alphabet Letter via
is used for Gesture Recognition. Using PCA we Hand Gesture”, International Conference on Computational
Science and Technology (ICCST), 2014.
calculate Eigen vector and feature vector to find
Euclidean distance. Minimum Euclidean distance
leads to character recognition for the input gesture.
This project helps special persons to communicate