2021 3rd International Conference on Signal Processing and Communication (ICPSC) | 13 – 14 May 2021 | Coimbatore

An Efficient Approach for Interpretation of Indian

Sign Language using Machine Learning
2021 3rd International Conference on Signal Processing and Communication (ICPSC) | 978-1-6654-2864-4/20/$31.00 ©2021 IEEE | DOI: 10.1109/ICSPC51351.2021.9451692

Dhivyasri S Krishnaa Hari K B Akash M

Department of ECE Department of ECE Department of ECE
PSG College of Technology PSG College of Technology PSG College of Technology
Coimbatore, India Coimbatore, India Coimbatore, India
dhivyasri2000@gmail.com krishnaaharikbk@gmail.com akashmohan1999m@gmail.com

Sona M Divyapriya S Dr. Krishnaveni V

Department of ECE Department of ECE Department of ECE
PSG College of Technology PSG College of Technology PSG College of Technology
Coimbatore, India Coimbatore, India Coimbatore, India
sonamani05121998@gmail.com divyapriyasivam98@gmail.com vk.ece@psgtech.ac.in

Abstract—Non-verbal communication involves the usage of

Sign Language. The sign language is used by people with hearing
/ speech disabilities to express their thoughts and feelings. But
normally, people find it difficult to understand the hand gestures
of the specially challenged people as they do not know the
meaning of the sign language gestures. Usually, a translator
is needed when a speech / hearing impaired person wants to
communicate with an ordinary person and vice versa. In order to
enable the specially challenged people to effectively communicate
with the people around them, a system that translates the Indian
Sign Language (ISL) hand gestures of numbers (1-9), English
alphabets (A-Z) and a few English words to understandable text
and vice versa has been proposed in this paper. This is done using
image processing techniques and Machine Learning algorithms.
Different neural network classifiers are developed, tested and
validated for their performance in gesture recognition and the
most efficient classifier is identified.
Index Terms—Indian Sign Language, hand gestures, inter-
preter, SURF, Convolutional Neural Network, Recurrent Neural
Network, K-means clustering, Support Vector Machine
I. I NTRODUCTION
Sign Languages vary throughout the world. There are
around 300 different sign languages used across various parts
of the world. This is because sign languages were developed
naturally by people belonging to different ethnic groups.
Perhaps, India does not have a standard sign language. Lexical
variations and different dialects of Indian Sign Language are
found in different parts of India. But, recently, efforts had
been taken to standardize the Indian Sign Language (ISL).
The ISL hand gestures are divided into two broad categories:
(i) static gestures, and (ii) dynamic gestures. The static ISL
hand gestures of numbers (0-9), English alphabets (A-Z), and
some English words are shown in Fig. 1.
According to the 2011 census, there are around 50 lakh people
in India with speech/hearing impairments. But, there are only
less than 300 educated and trained sign language interpreters
in India. So, people with speech/hearing impairments tend to
Fig. 1. ISL Hand Gestures
become isolated and lonely, as they face difficulties in com-
municating with other normal people. This has a tremendous
effect on both their social and working life.
Due to the above mentioned challenges that the specially
challenged people face, an automated real-time system that
could translate English words to ISL and vice versa has
been proposed in this paper. This system makes it easy for
the specially challenged people to communicate effectively
with the rest of the world. This could enhance their abilities
and make them realize that they can do better in life. The
proposed system performs two major tasks: (i) Gesture to
Text conversion and (ii) Speech to Gesture conversion. Gesture
to text conversion is done using neural network classifiers.
Speech to gesture conversion is done using Google Speech
Recognition API.
This paper focuses on conversion of standard Indian Sign
Language gestures to English, and conversion of English
words (spoken) to Indian Sign Language gestures with highest
possible accuracy. For this, different neural network classifiers
are developed, and their performance in gesture recognition
is tested. The most accurate and efficient classifier is chosen
and is used to develop an application that converts ISL
gestures to their corresponding English text, and speech to

978-1-6654-2864-4/21/$31.00 ©2021 IEEE 130

Authorized licensed use limited to: San Francisco State Univ. Downloaded on July 02,2021 at 19:06:28 UTC from IEEE Xplore. Restrictions apply.
 2021 3rd International Conference on Signal Processing and Communication (ICPSC) | 13 – 14 May 2021 | Coimbatore
the corresponding ISL gestures.
II. P ROPOSED M ETHODOLOGY
As mentioned in the above section, the proposed system
for ISL interpretation performs two major tasks: (i) Gesture
to Text conversion and (ii) Speech to Gesture conversion.
A. Gesture to Text Conversion
Gesture to text conversion involves four major steps: (i)
Dataset collection, (ii) Segmentation, (iii) Feature Extraction
and (iv) Classification. The concept diagram of gesture to text
conversion is given in Fig. 2. The first step in gesture to text
Fig. 2. Concept Diagram - Gesture to Text Conversion
conversion is dataset collection. An image dataset consisting
of ISL hand gestures of 9 numbers (1-9), 26 English alphabets
and a few English words is collected. After the dataset is
ready, all the images in the dataset are pre-processed to mask
the unwanted areas and to remove noise from the image.
Hence, pre-processing the images prior to feeding them to
a classifier improves the efficiency, accuracy and performance
of the system. Hence, this step is very important in the image
classification process.
Data pre-processing involves the following steps:
1) Resizing the images to the same size (for uniformity)
2) Conversion of RGB image to Grayscale image
3) Median Blur
4) Skin Masking and Skin Detection
5) Canny Edge Detection (to detect sharp edges in the
image)
Next step in Gesture to text conversion is feature extraction.
Feature extraction is done on the pre-processed images.
Feature extraction is a very important step in computer vision
and image classification processes. This involves converting
the raw data (images) into numerical features so that the data
can be processed by the classification algorithm. Though the
image is converted into numerical form, the information in
the original data is preserved.
131
Authorized licensed use limited to: San Francisco State Univ. Downloaded on July 02,2021 at 19:06:28 UTC from IEEE Xplore. Restrictions apply.
Here, feature extraction is done using the Speeded-Up
Robust Feature (SURF) method. The SURF is used as a
feature descriptor or as a feature detector. It is often used
for applications like object detection, image classification
etc. It is a fast and robust algorithm for representing and
comparing images. It acts as a blob detector in an image. The
SURF features are calculated by finding the interest points
in the image that contain the meaningful features using the
determinants of Hessian matrices. For each interest point found
in the previous process, the scale invariant descriptors are
constructed.
The Hessian matrix and its determinant are given in (1) and
(2) respectively.
 2 2 
∂ f ∂ f
∂x2 ∂x∂y
H(f (x, y)) =  ∂2f ∂2f
 (1)
∂x∂y ∂y 2
det(H) = Dxx Dyy − (0.9Dxy )2 (2)
The extracted image features are fed as input to different
Machine Learning algorithms like Convolutional Neural Net-
work (CNN), Recurrent Neural Network (RNN) and Support
Vector machine (SVM).
SVM is a supervised machine learning algorithm that uses a
hyper-plane to separate different classes of data. The SVM
classifier is used along with K-means clustering classifier and
Bag of Visual words (BoV) model to achieve better accuracy.
The K-means clustering classifier is an unsupervised classifier
that is used to group the similar data points to ‘k’ number
of clusters, where ‘k’ is the number of classes in the dataset.
The output of the k-means clustering model is fed to the BoV
model for classification. The BoV model further classifies the
images based on the count of visual words (image features
are considered as visual words) that occur in an image. The
output of the BoV classifier is fed to the SVM. The SVM is
the main classifier with which training and testing are done.
In the SVM classifier, about 80% of the data in the dataset is
used for training and the remaining 20% of the data is used
for testing.
The CNN and RNN classifier models are designed, and their
performance in gesture recognition is noted. For both CNN
and RNN, the data in the dataset is divided into three parts:
(i) 60% of the data is used for training, (ii) the next 20% of
the data is used for testing and (iii) remaining 20% of the data
is used for validation.
The performance of all the above mentioned classifiers is
validated to figure out the most efficient image classifier for
gesture recognition.
The classifier with the highest accuracy is identified and then
used to detect ISL gestures in a live video. Recognizing the
gestures in live video is as simple as identifying the gestures
in a still image. The steps involved in identifying the gestures
in live (real-time) video are listed below.
1) Video is captured using Web Camera
2) Each frame in the video is captured as an image
 2021 3rd International Conference on Signal Processing and Communication (ICPSC) | 13 – 14 May 2021 | Coimbatore

3) The captured images are resized and pre-processed

4) SURF Features are extracted
5) The image features are passed to the classifier
6) Gestures are predicted

B. Speech to Gesture Conversion

The conversion of Speech to ISL gestures is done using the
following processes:
1) Conversion of Speech to Text
2) Comparing the text obtained from the previous step with
the database Fig. 4. Data pre-processing
3) Displaying the corresponding ISL Gesture Output
Speech to text conversion is done using PyAudio and Google
Speech Recognition API. The concept diagram of speech to
gesture conversion is shown in Fig. 3.

Fig. 5. SURF Features

1) Support Vector Machine: The input images were passed

Fig. 3. Concept Diagram - Speech to Gesture Conversion
III. E XPERIMENTAL R ESULTS
A. Gesture to Text Conversion
A dataset consisting of ISL hand gestures of numbers (1-
9), English alphabets (A-Z), and 7 English words (BOAT,
FRIEND, HOLIDAY, OK, SWING, SMILE and STAND) was
collected (refer Fig. 1). So, the dataset consisted of 42 (9 + 26
+ 7) classes of images. 1200 different images were captured
for each image class in the dataset.
The images in the dataset were pre-processed to mask un-
wanted areas in the image and to remove noise as mentioned
in the previous section. The various image pre-processing steps
performed on a sample image from the dataset are shown in
Fig. 3.
The SURF feature matrix was calculated for every image
in the dataset after pre-processing them. The SURF features
extracted for a sample image is shown in Fig. 4. The blue
colored circles of varying sizes found in Fig. 4 are the
SURF Feature points. The SURF features of all the images
were extracted and stored in a pickle file and then fed into
different neural network classifiers. The accuracy of each of
the classifiers tested is discussed below.
to the K-means clustering and Bag of Visual words classi-
fiers before passing them to the SVM classifier. As there
are 42 classes of images in the dataset, k=42 for the K-
means clustering classifier. The visual words were collected
for both the test and train dataset after applying the k-means
clustering algorithm. There are totally 50391 images in the
dataset. Among these images, 40320 images were used for
training the SVM model. The remaining, 10071 images were
used for testing the performance of the classifier model. A
testing accuracy of around 99.5% was achieved. The other
performance metrics like precision score, F1 score, and recall
score were also calculated. These are shown in Fig. 6.
Fig. 6. Accuracy of SVM Classifier
2) Convolutional Neural Network: A Convolutional Neural
Network was modeled and developed using the Keras library
in Python. Around 30,240 images (60% of the images in the
dataset) were used to train the classifier model. The classifier
was trained with different number of epochs. A maximum
average testing accuracy of around 88.89% was obtained.
3) Recurrent Neural Network: A Recurrent Neural Net-
work was modeled and developed using the Keras library in

132

Authorized licensed use limited to: San Francisco State Univ. Downloaded on July 02,2021 at 19:06:28 UTC from IEEE Xplore. Restrictions apply.
 2021 3rd International Conference on Signal Processing and Communication (ICPSC) | 13 – 14 May 2021 | Coimbatore
Python. Around 30,240 images were used to train the classifier
model. The classifier was trained with different number of
epochs. A maximum overall testing accuracy of around 82.3%
was obtained.
From the above results obtained, it is concluded that the
combination of K-Means clustering, BoV, and SVM classifiers
has the highest accuracy in recognizing the hand gestures.
Hence, it is more reliable for gesture recognition.
B. Gesture Recognition in Live video
A real-time gesture recognition system was developed using
the SVM classifier. When the user shows an ISL hand gesture
in front of the camera, the corresponding English text is
displayed. The time taken to predict the hand gesture in the
real-time video is about 0.04s. Fig. 7 shows the screenshots
of the real-time gesture recognition system.
Fig. 7. Real-time gesture recognition
C. Speech to Gesture Conversion
Speech to gesture conversion is done using Google Speech
Recognition and PyAudio. As Google Speech Recognition API
is used, this process requires an active internet connection.
The speech duration is set to 5 sec. i.e. the user is given 5
seconds to speak out the word into the microphone. After
this, the Google Speech Recognition API converts the speech
into text. Then, the ISL gesture corresponding to the predicted
text is displayed. The implementation screenshot of Speech to
Gesture conversion of the word ”Hello” is given in Fig. 8.
Fig. 8. Speech to Gesture Conversion
133
Authorized licensed use limited to: San Francisco State Univ. Downloaded on July 02,2021 at 19:06:28 UTC from IEEE Xplore. Restrictions apply.
IV. C ONCLUSION
From the results obtained, it is inferred that the SVM clas-
sifier along with the K-means clustering and BoV classifiers is
best suited for gesture recognition. A user friendly application
that can interpret Indian Sign Language has been developed
using the most efficient SVM classifier (for gesture to text
conversion) and Google Speech Recognition API (for speech
to gesture conversion). Thus, a more reliable sign language
interpretation system has been developed.
ACKNOWLEDGMENT
We would like to thank PSG College of Technology,
ECE Department for providing an opportunity to work
on the project. We thank Dr. V. Krishnaveni, Professor
(CAS) & Head In-charge, Department of Electronics and
Communication Engineering for the encouragement and
support that she extended towards this project work. We also
thank our Programme Coordinator Dr. M. Santhanalakshmi,
Associate Professor, Department of ECE and our Tutor Ms.
P. Prabavathi, Assistant Professor, Department of ECE for the
advice and constructive feedback regarding various aspects
of the project.
R EFERENCES
[1] Cheok Ming Jin, Zaid Omar, Mohamed Hisham Jaward, ”A Mobile Ap-
plication of American Sign Language Translation via Image Processing
Algorithms”, in IEEE Region 10 Symposium, on IEEEexplore, 2016.
[2] Mr. Sanket Kadam, Mr. Aakash GhodkeProf. Sumitra Sadhukhan, ”Hand
Gesture Recognition Software based on ISL”, IEEE Xplore, 20 June
2019.
[3] Kartik Shenoy, Tejas Dastane, Varun Rao, Devendra Vyavaharkar, ”
Real-time Indian Sign Language (ISL) Recognition”, IEEE Xplore: 18
October 2018.
[4] T Raghuveera, R Deepthi, R Mangalashri And R Akshaya, “A depth
based ISL Recognition using Microsoft Kinect”, ScienceDirect, 2018.
[5] Muthu Mariappan H, Dr Gomathi V, “Real time Recognition of ISL”,
IEEE Xplore: 10 October 2019.
[6] G. Ananth Rao a, P.V.V. Kishore, “Selfie video based continuous Indian
sign language recognition system”, ScienceDirect, 2018.
[7] G. Ananth Rao a, P.V.V. Kishore, “Sign Language Recognition Based
On Hand And Body Skeletal Data”, IEEE 2018.
[8] Suharjitoa, Ricky Andersonb, Fanny Wiryanab, Meita Chandra Ariestab,
Gede Putra Kusuma, “Sign Language Recognition Application Systems
for Deaf-Mute People: A Review Based on Input-Process-Output”,
ScienceDirect, 2017.