An Assistive Hand Glove for Hearing

and Speech Impaired Persons

Marvin S. Verdadero, Jennifer C. Dela Cruz
School of Electrical, Electronics and Computer Engineering
Mapúa University
Manila, Philippines
msverdadero@biscast.edu.ph

Abstract — Communication is more than spoken language; it
is a way of expressing ones thought, ideas and messages.
Communication for them has become a great challenge especially
the need to communicate with normal people in their daily
routine. Hearing/Speech Impaired persons used hand sign
language as their language of communication; however, it will
only be effective if the person they are interacting with knew the
hand sign language. This study developed an assistive device for
persons with hearing/speech impairment. The system was
designed to interpret their signs or gesture into a corresponding
message which were preloaded in the device. The device
produced a voice-based message through the audio module and
displays the message in LCD. This study was basically a glove
based system and a microcontroller based system. Flex sensors,
Gyroscope and Accelerometer were incorporated in a glove that
can identify the sign language and an Arduino Nano
microcontroller unit converted these gestures into voice and text
output. The result of the Assistive device during its test-
experiments showed that the system can recognized the hand
gestures with male and female voice output on each hand gesture
with an overall accuracy of recognition of 83.58%.
Keywords: Assistive Device, Glove-Based System, Voice Based
Announcement, Gesture – Based Sensors, Hand Gesture
I. INTRODUCTION
In the Philippines, the results of the 2010 Census of
Population and Housing (CPH, 2010) showed that of the
household population of 92.1 million, 1.443 million Filipinos
or 1.57%, have a disability [1] and these disabilities include
Hearing and Speech Impaired persons. Communication plays
an important role to human life. One should learn how to
communicate his thoughts, emotions and reactions for it to be
understood by others. Communication is the basic method in
order to overcome the distance and gap built between
individuals, and to develop relationships with others. It is a
learned skill. It is important to have a common medium in
order to interpret and deliver a message and vice versa [2].
Today, modern technology has made advances on how people
communicate. Countless inventions and innovations have been
designed and developed by many great minds whose drives
were mainly to make people’s lives way better than before
technological advancements paved way. It started out
as simple tools made by the earliest men and evolved into
wireless technologies, complex machineries and gadgets
which made the necessity to do their tasks quickly and to
communicate effectively. Since this kind of technology was
introduced to people, it strengthens the rapid changes and
triggers the endeavor of a more enhanced knowledge to
overcome the incremental demand for its uses. It is obvious
how technology change the lives of people. But more than
that, such technology is profoundly life-changing for people
with needs, such as those with disabilities, giving them the
opportunity to do things they aren’t capable of or was capable
of, and making it possible for them to do so again[3]. Human
beings are blessed with the ability to see, listen and speak to
communicate. However, there are those who are deprived of
these valuable gifts, making it hard for them to communicate
normally. Some have physical disabilities that were inborn or
acquired in an unfortunate circumstance. They are using signs
as their language of communication and will only be effective
if the person interacting with them knew how to understand
such. Not all people can communicate well as what others can.
Communication for people with hearing/speech impairment
has become a challenge for them to communicate normally in
their daily routine [4]. Sign language is a universal tool for the
speech and hearing impaired. Different countries have their
own recognized official sign language. For instance, Indians
have their Indian Sign Language (ISL), Americans have
American Sign Language (ASL), and Filipinos use the Filipino
Sign Language (FSL). FSL belongs to the branch of visual
languages influenced by American Sign Language together
with Thai Sign Language and Kenyan Sign Language to name
a few. However, the structure of FSL has changed
significantly enough for it to be considered a distinct language
from American Sign Language [5]. Since communication is
the fundamental aspect of human society, numerous forms of
assistive devices are now becoming available to help those
with hearing and speech impairment to communicate and live
normal lives. These are the “assistive devices”. It serves as a
communication medium for those with hearing/speech
impairment and normal people [6]. One of such
communication aid devices is the “Glove Based System”.
Several gesture - based sensors are placed on gloves that are
worn on the handicapped user’s hand. The glove’s sensors will
sense and define the hand gesture through sign language that
corresponds to a specific message. When the sensor detects the
hand gestures, voice and texts are produced indicating what
the handicapped person wanted to say [7].

978-1-7281-3044-6/19/$31.00 ©2019 IEEE

 This assistive device is a microcontroller based system
designed to understand and define the signs which were pre-
loaded in the device. It is making use of gesture-based sensors
to define the hand gesture of the user, indicating the message
they would want to convey. This will produce a voice-based
output and display the message on the LCD for the user to
read and confirm what the device will convey to the other
person. Moreover, as most of the prevailing systems have used
the American Sign Language (ASL) and its alphabet as its
basis for the input data, this device uses the analysis of
Filipino Sign Language (FSL) alphabet for which every sign
corresponds to a specific message. FSL alphabet is used as the
device is intended for Filipinos with hearing/speech
impairment.
This device can be utilized by the persons with
hearing/speech impairment as they attend school or socialize
in public and connect with others. This system makes things
possible for them to have a voice and be heard.
The objective of this study is to design an assistive device
for people with hearing/speech impairment that interprets hand
sign languages into a readable text messages and recognizable
voice audio.
This research will make it easier for people with
hearing/speech impairment to convey message and reduces the
communication gap as well as reduces the need to accompany
a handicap in their daily routine.
This research should produce an assistive device designed
in a glove fitted with gesture-based sensors to communicate. It
will detect and define sign language made by the user to give
the corresponding message programmed in the microcontroller
through voice output produced by a speaker and text output
displayed in LCD. The device has its slide switch for male or
female voice selection to produce a voice output depending on
the user’s gender.
This device is not suitable for people with hearing/speech
impairment that have problems with finger movements. It can
also be used by paralyzed people who can still properly use
their fingers. The user should be knowledgeable in sign
language matching the specific message and can only use the
signs applicable as defined in the input data which mainly
focuses on the common thoughts and needs of people with
speech and/or hearing impairment. This device will analyze
the hand gestures made as defined in the Filipino Sign
Language alphabet with the input data being fed in the
device’s controller. The user must be familiar with the
corresponding message for every gesture programmed in the
device.
Once the device is in use, proper movement of fingers
should be controlled to avoid having an inaccurate definition
of the hand gesture. Moreover, the glove is only designed for
right-handed person.
The device has assigned a corresponding message for each
of the alphabet letters from A – Z, except for J and Z, as it
requires certain movements for a specific sign language.
II. RELATED WORKS
The following studies presented play a very important role
on the development of the research.
In paper [8] was a glove-based system aimed to develop a
computerized Indian Sign Language (ISL) recognition system.
The designed system input hand gestures to the system
through an electronic sensor glove and it identifies the gesture
patterns via a microcontroller network. Then the identified
sign will be converted to text and then translated to voice
output. Data glove consisted of 5 potentiometer used as
sensors which measure the bending of the five fingers.
Based on the paper [9] designed to overcome the difficulty
of deaf-mute people while talking to other people. An
accelerometer sensor was used as a tilt sensor that played the
significant part. The system made use of 2 three axes
accelerometer sensors which detects the change in resistance
depending on the movement of hand. This hand movement
was sensed by the microcontroller which in turn checks the
data and speaks out the prerecorded message assigned for that
particular movement through the speaker and displayed on
LCD.
According to [10] is a speech recognition system with
embedded controllers and audio pre recorder which provides
help to deaf-mute people to express their need to the normal
people. This glove design requires an AT89S52
microcontroller interfaced with flex sensors to capture the
words. The words or signs were obtained with the English
language as reference. The microcontroller was placed to
process finger movements to verbal audio, thus the flex sensor
and microcontroller values change consequently, and by
comparing these two values, the output is displayed and the
voice module gives the corresponding audio output.
The paper presented [11] used an Atmega 328 and sensors
based interacting device. This study analyzed the data from an
instrumented data glove for use in recognition of some signs
and gestures and their conversion into speech. The data glove
was equipped with two types of sensors: the bend sensors,
made by using 555 timer IC along with a phototransistor, and
accelerometers as tilt sensors.
Researchers continuously develop and working in the field
of gesture recognition. The applications of glove based system
were popular to deaf-mute people but have extended to the
benefit of physically challenged people. The additional feature
developed was the user can remotely turn-off other electronic
and electrical devices.
In addition [12] discussed how a user wore gloves fitted
with flex sensors whose resistances change with each gesture.
This produced a voltage change and the PIC microcontroller
used will display the codes corresponding to each gesture on
the LCD and the sound code will be heard via the speaker.
Also, this device helped to remotely turn on or off other
devices like fan, bulb etc. by means of RF transmitters and
receivers. A wireless camera fitted near the door enables
image recognition and automatic control to open or close it as
the user wish to. The system could also send an SMS to a pre-
stored mobile number via a GSM modem in case of an
emergency.
More so [13] presented a device made use of an
accelerometer to measure the speed when the user moved his
hand or finger. The heartbeat sensor and temperature sensor
were also used to sense the user all day. The output was
predefined by a voice driver circuit generated from the PIC,
the microcontroller used in the system. If any change occurs, it
will warn the attender by using the GSM through the
corresponding mobile phone number.
 The above gesture recognition systems were all sensors
based, but an alternative presented [14] a device that converts
gestures of the user into verbal audio. The system uses image
processing technology and neutral networking for the
capturing and conversion of gestures. Image acquisition was
the basic process in this device. It comes with a web camera to
capture the hand gestures shown by the user. The image
captured is progressed in the image processing toolbox in the
MATLAB and matched in the database to generate the
corresponding verbal audio.
Aside from glove – based systems, another wearable
communication method presented [15] wherein a
microcontroller based system designed to make convenient for
the deaf and mute to call for help. In this study, Arduino nano
boards, NRF transceiver, AT mega 328, GSM module based
interacting system was presented. The device has used two
arduino nano bands (one for the deaf – mute and the other for
the hearing impaired). This system is consisted of two
wearable wrist bands, each to be worn by the patient and his
attendant. The patient’s band contained three switches to be
pressed, with each depending on the need. While the
attendant’s band consists of a buzzer that beeped as the patient
pressed a certain switch and an LCD showing the name of the
patient calling and his requirement. Therefore, it is made
easier for the attendant to aid the deaf and mute person.
III. METHODOLOGY
which shows the message being announced. The entire device
was driven by power bank that acts as its power source.
Electronic Components
The electronic parts used in constructing the project were
stated, namely: (A) Main Controller, (B) Sensors and (C)
Output Modules.
A. Main Controller
The Arduino Nano as the microcontroller is considered as
the brain of this device. This controls the entire operation. The
researchers chose to use Arduino Nano, the microcontroller
board based on the ATmega328.
B. Sensors
Out of the many sensors used by different existing systems,
the researchers used of flex sensors, a gyroscope and
accelerometer that are responsible for detecting the
movements in the user’s hand that wore the glove.
The device used five flex sensors that were mounted on
each finger in the glove. Three simple flex sensors of 2.2
inches and two 4.5 inches in length were used in the design.
As the sensors are flexed, the resistance across the sensor
varies. The combined 3-axis gyroscope and a 3-axis
accelerometer were used for the orientation and movement of
the hand. Flex Sensors and gyroscope and accelerometer were
fitted in the glove. These sensors sense the hand gestures with
varying resistance and according to the angle movement of the
fingers. This system used the Filipino Sign Language Alphabet
as shown in Figure 2 as basis for giving the corresponding
message to every letter.

Fig. 1. Block Diagram of the Device

Figure 1 shows the Block Diagram of the device. The main

component was the microcontroller unit, Arduino Nano with
an inbuilt Atmega328 which served as the main control for the
system.
Next components were the gesture – based sensors that
were fitted with a glove: flex sensors and gyroscope and
accelerometer. These sensors were responsible for detecting
the hand gestures as input being fed in the controller. Five flex
sensors were fitted for each finger and measure the angle of Fig. 2. Filipino Sign Language Alphabet
bending of fingers measured. When the user bends his finger
the sensor produces output resistance in relation with the
bending angle. Gyroscope and accelerometer was employed as C. Output Modules
a tilt sensing element that checks the position of the hand. The MP3-TF 16P module was used for the voice–based
Audio module and LCD were the modules that give the output that uses a memory card to store voice recordings and a
desired output generated from the microcontroller. The MP3 16x2 LCD for the display of the interpreted gesture from the
module with the simplified output speaker produces the voice device.
output. It is used so that speech codes corresponding to each
gesture can be initially recorded and played back when
identified by the controller. The LCD gives a text output
 IV. RESULTS AND DISCUSSION W 43 77 27 27 18
X 43 54 10 13 16
Figure 3 shows the design of the device and Figure 4 shows
the placement of the different electronic components such as Y 77 36 14 16 33
the main controller, sensors, and output modules used in the Hi 76 75 25 24 30
design of the device. It’s Hard 39 63 17 13 17
Okay 79 34 10 12 17

Table 1 shows the different calibration values of the five

Fig. 3. Design of the Device
The components were placed according to the design and
were connected in their appropriate connection before
conducting experiments to have a precise and proper
calibration of the sensors.
(5) flex sensors used in the design. Flex Sensor 1 mounted on
Thumb Finger, Flex Sensor 2 mounted on Index Finger, Flex
Sensor 3 mounted on Middle Finger, Flex Sensor 4 mounted
on Ring Finger, and Flex Sensor 5 mounted on Pinky Finger.
Each flex sensor calibration value was based on its bending
when making a gesture. Flex Sensors calibrations were done to
get the minimum and maximum value from a sensor to help
find its operation range at different hand gestures to convey
the different pre-programed messages of the device.
Table 2. Gyroscope and Accelerometer Calibration Values
Sign
Language Coordinates Right Left Up Down

x 1316 2752 9864 -1532

H y 15688 15888 12952 15640
z -4412 -3248 -3804 -3584
x -4176 6648 7528 -6332

P y 15072 14788 14912 14492

z -4368 -1556 -1544 -980

x -5868 -5628 -2744 -12576

Q y 14580 15112 15812 8316

Fig. 4. Placement of the Components
z 536 1950 1232 3072
Table 1. Flex Sensor Calibration Values
Sign Flex Sensor
Table 2 shows the different Gyroscope and Accelerometer
Language 1 2 3 4 5
Calibration values for the Letters H, P, and Q. These sensors
A 51 38 12 13 15 were employed to assess the tilting and orientation of the
B 47 84 28 30 33 device around x, y and z-coordinates. These values were used
C 51 60 16 17 24 to distinguish letters which had the same and nearer flex
D 56 69 15 17 23 sensors calibration values with other letters when gestured.
E 41 47 10 11 13
F 51 49 24 27 29 Table 3. Phrases and Hand Gesture
G 59 58 12 13 16 Audio Length
File Letter Hand Gesture
Phrase
H 40 64 25 14 17 Name Assigned Female Male (Gesture Assigned)

I 41 36 12 15 31 School
K 64 68 24 15 16
L 76 71 11 15 17 Thank you 0001 0028 0.86s 0.78s
M 34 37 14 13 17
N 38 40 15 13 18
O 57 48 16 18 24 You’re
0002 0029 0.97s 1.04s
Welcome
P 56 63 26 15 17
Q 70 53 13 14 16
R 40 66 22 15 18 Goodbye
Teacher! I’m 0003 0030 2.90s 2.48s
S 39 40 10 13 16 going home.

T 47 46 11 17 18
U 50 70 26 17 22
I’m Sorry 0004 0031 0.99s 0.91s
V 43 76 26 15 18
 Standard
Good Day 0005 0032 0.91s 0.91s

HI! 0025 0052 0.6s 0.50s

Excuse Me 0006 0033 1.10s 1.02s

It’s Hard. 0026 0053 0.97s 0.86s

I’m not
0007 0034 1.62s 1.70s Okay 0027 0054 0.78s 0.73s
feeling well

I need to go
to CR
0008 0035 1.93s 1.93s Table 3 shows the phrases that were programmed into the
system, with each phrase file name stored in the memory card.
The system required filenames with 4-digit numbers. Also
Home
included in the table is the corresponding letter per phrase and
I’m home 0009 0036 0.81s 0.91s
the audio length of each phrase. The researchers included three
standard sign languages that persons with hearing/speech
impairment are also using.
Can you get
me some 0010 0037 1.80s 1.59s Table 4. Experimental Results and Analysis of the Assistive Device
water?
Sign Did Not
Recognized
Language Recognized
Is there a
good Frequency 23 7
program in
0011 0038 2.46s 2.29s A
Percent 76.7 23.3
TV?
Frequency 25 5
Can you B
please pass Percent 83.3 16.7
0012 0039 2.69s 2.48s
the remote Frequency 28 2
control? C
Percent 93.3 6.7
I’m already Frequency 26 4
0013 0040 1.41s 1.36s D
full Percent 86.7 13.3
Frequency 28 2
E
Percent 93.3 6.7
I am fine 0014 0041 1.02s 1.02s
Frequency 29 1
F
Percent 96.7 3.3
Frequency 26 4
I’m hungry. G
Can we eat?
0015 0042 2.22s 2.14s Percent 86.7 13.3
Frequency 28 2
H
Percent 93.3 6.7
I am going to
my classmate
0016 0043 1.85s 1.96s Frequency 24 6
I
Percent 80 20
Frequency 20 10
Social K
Percent 66.7 33.3
What is your Frequency 19 11
0017 0044 1.20s 1.10s L
name? Percent 63.3 36.7
Frequency 28 2
M
Percent 93.3 6.7
Bless you 0018 0045 1.04s 1.04s
Frequency 22 8
N
Percent 73.3 26.7
That helps a
lot. Thank 0019 0046 2.56s 2.35s Frequency 23 7
O
You! Percent 76.7 23.3
Frequency 27 3
P
What do you
0020 0047 1.28s 1.36s
Percent 90 10
mean?
Frequency 24 6
Q
Percent 80 20
I’m about to Frequency 29 1
0021 0048 2.43s 2.31s R
leave. Bye! Percent 96.7 3.3
Frequency 18 12
S
Percent 60 40
How much
0022 0049 1.57s 1.72s Frequency 28 2
does it cost?
T
Percent 93.3 6.7
Frequency 20 10
What are you U
0023 0050 1.18s 1.20s Percent 66.7 33.3
doing?
Frequency 30 0
V
Percent 100 0
I want to go
Frequency 23 7
0024 0051 1.46s 1.57s W
to mall. Percent 76.7 23.3
X Frequency 26 4
 Percent 86.7 13.3 ACKNOWLEDGMENT
Frequency 29 1
Y
Percent 96.7 3.3 The authors would like to thank all those people who
Hi
Frequency 18 12 helped and contributed their insights and suggestions to the
Percent 60 40 realization of this research.
Frequency 27 3
It’s Hard
Percent 90 19 REFERENCES
Frequency 29 1
Okay [1] “Persons with Disability in the Philippines (Results from the 2010
Percent 96.7 3.3
Census),” Philippine Statistics Authority, 10-Jan-2013. [Online].
Over All Frequency 677 133
Available: https://psa.gov.ph/content/persons-disability-philippines-
Accuracy Percent 83.58 16.42
results-2010-census. [Accessed: 26-Jan-2019].
[2] Speak, H. (2014). Why is communication important to human life? -
Table 4 shows the Experimental Results and Analysis of Hope Speak. [online] Hope Speak. Available at:
https://www.hopespeak.com/blog/why-is-communication-important-to-
the Assistive Device conducted on the different hand sign human-life-2/ [Accessed 17 Dec. 2018].
languages. Experiments were conducted in 30 trials for each [3] Radabaugh M.P. (2015). Assistive Technology: Independent Living
sign language to measure the ability of the Assistive Device in CentreNSW.Available at: https://www.ilcnsw.asn.au/home/
terms of recognizing the different hand gestured with a assistive_technology/assistive_technology.html
[4] P. Lokhande, R. Prajapati, and S. Pansare, “Data Gloves for Sign
readable text message and recognizable voice audio as its Language Recognition System,” Int. J. Comput. Appl., pp. 975–8887,
output. The experiments cover the 24 Filipino Sign Languages 2015.
and three 3 standard sign languages with male and female [5] Martinez, L. (2012). Primer on Filipino Sign Language. [online]
voice output. It also shows the frequency of recognized and Opinion.inquirer.net. Available at:
https://opinion.inquirer.net/41909/primer-on-filipino-sign-language
did not recognized sign languages with corresponding [Accessed 17 Dec. 2018].
percentage to determine its accuracy of recognition on each [6] NIDCD Information Clearinghouse, “Assistive Devices for People with
hand gesture performed by the Hearing/Speech Impaired Hearing, Voice, Speech, or Language Disorders | NIDCD,” Natl.
Persons. Institutes Heal., pp. 1–6, 2011.
[7] Sunitha, K., Saraswathi, A., Aarthi, Jayapriya, & Sunny, L. "Deaf Mute
Communication Interpreter - A Review". International Journal of
Applied Engineering Research, pp. 290-296, 2016.
[8] Patel, B., Shah, V., & Kshirsagar, R. "Microcontroller Based Gesture
Recognition System For The Handicap People". Journal of Engineering
Research and Studies, pp. 113-115.2011.
[9] Nikam, M. T., Ghodunde, S. R., Sonawane, K. V., & Gawade, V. R.
"Talking Hands for Deaf and Dumb People". International Journal of
Scientific Engineering and Technology Research, pp. 565-568.2015.
[10] Mertiya, D., Dadhich, A., Verma, B., & DipeshPatidar. "A Speaking
module for Deaf and Dumb". International Journal of Electronics and
Communication Engineering, pp.78-80.2016.
Fig. 5. Actual Device
[11] Sondhi, A., Kasa, P., & Solanki, K. "Interacting Device for Deaf &
Dumb using Atmega 328 Processor". International Journal Of
Figure 5 shows the actual image of the device showing the Engineering And Computer Science, pp.14439-14442.2015.
different components mounted and connected to one another to [12] Dinesh, Sruthi. "Talking Glove- A Boon for the Deaf, Dumb and
Physically Challenged". International Journal of Advanced Research in
the glove. Electronics and Communication Engineering, pp.1366-1369.2015.
Effectiveness of the Assistive Device [13] Sabrigiriraj, M., Nirmal Krishna, R., Arun Kumar, S., & Aashif, A. "An
Electronic Support System for Dumb and Paralyzed Person". Asian
The accuracy and effectiveness of the Assistive Device Journal of Applied Sciences and Technology, pp. 277-278.2017.
were attained by numerous testing and experiments for its sign [14] Abraham, N., George, N., Rwithvik, K., & Jisha, K. "Dumb Aid
Phone". International Journal of Emerging Technology and Advanced
language recognition and output. The device’s response was
Engineering, pp. 524-527.2015.
tested for the sign languages with a male and female voice [15] Gupta, M., Abrol, S., Anand, T., Verma, H., & Raina, G. "AVR and
output for 30 trials. ARM Based Speaking System For Deaf and Dumb". International
Experiments showed that the design of the Assistive Journal of Advanced Research in Science, Engineering and
Technology, pp. 3814-3816.2017.
Device for people with hearing/speech impairment can
[16] Fromtexttospeech.com. (2018). From Text To Speech - Free online
recognize and interpret the different hand gestures performed TTS service. [online] Available at: http://www.fromtexttospeech.com/.
for both male and female voice output with an overall
accuracy of 83.58% in terms of recognition.

V. CONCLUSION
The design of An Assistive Device for Hearing and Speech
Impaired Persons can be used as a medium for them to express
their thoughts, ideas, and messages on the people they
encounter on their everyday lives.
The researchers concluded that the device works
accordingly as it translates sign languages into readable texts
messages and recognizable voice audio with an overall
accuracy of 83.58% in terms of recognition.