FEDERAL UNIVERSITY OYE EKITI

PROJECT PROPOSAL

SIGN LANGUAGE RECOGNITION FOR DEAF AND

DUMB USING TENSORFLOW

NAME: AKINLADE JOSIAH AKINKUNMI

MATRIC NO: CSC/2017/1045

DEPARTMENT: COMPUTER SCIENCE

FACULTY: SCIENCE

LEVEL: 400

SUPERVISOR: MRS DARAMOLA

CHAPTER ONE

1.1 INTRODUCTION

Communication forms the foundation of all human relationship. It is an essential part of our daily
life. It is a process of creating, exchanging, sharing ideas, information, opinions, facts, feelings, and
experiences between a sender and a receiver. Communication is fundamental to the existence and
survival of individuals, groups, societies, and nations. Communication can be classified into two
forms, Verbal and non verbal. Verbal communication is through the words we speak and non-
verbal is through our actions and body language. However both verbal and
nonverbal communication can be spoken and written. Many people mistakenly assume that verbal
communication refers only to spoken communication.

Communication impaired people (deaf and dumb) are deprived of normal communication with
other people in the society. This may be due to some disabilities of organs responsible for speech
and vision; however does communication impairment means affected people can’t create exchange,
share ideas and information, No.

Sign Language became the principal substance of communication within the communication
impaired community. It has been observed that they find it really difficult at times to interact with
normal people with their gestures, as only a very few of those are recognized by most people. As
like any other language it has also got grammar and vocabulary but uses visual modality for
exchanging information. The trouble arises when communication impaired people try to express
themselves to other people with the aid of these sign language grammars. This is because people
are usually unaware of these grammars. As a result it has been seen that communication of a
communication impaired person are only limited within his/her family or his/her deaf-dumb
community. Hence there is a need of a system which recognizes the different signs, gestures and
conveys the information to the normal people. It bridges the gap between physically challenged
people and other people.

1.2 BACKGROUND OF THE STUDY

SIGN LANGUAGE
Sign is a mark or shape that always has a particular meaning, it give you information about
something,

A language is a structured system of communication used by humans. Languages can be based on

speech and gesture (spoken language), sign, or writing. The structure of language is
its grammar and the free components are its vocabulary.
A language is a system of conventional spoken, manual (signed), or written symbols by means of
which human beings, as members of a social group and participants in its culture, express
themselves.
A language is a system of communication which consists of a set of sounds and written symbols
which are used by the people of a particular country or region for talking or writing

Sign language (SL) are gestures made with the hands and other body parts, including facial
expressions and postures of the body. It used primarily by communication impaired people There
are many different sign languages as, Nigerian, British, Indian and American sign languages.
British sign language (BSL) is not easily intelligible to users of American sign Language (ASL)
and vice versa. Each gesture in the SL has its own logic, meaning and understanding to it. Learning
SL understands the meaning of the gestures. In fig 1.1 different alphabet sign languages are shown

Fig 1.1 Alphabet Sign Language

Gestures are the motion of the body or the physical action form by the human in order to convey
some meaningful information and to communicate. The gesture recognition is the process by which
gestures are made by the human and it’s made like its known to the system or the other human. It
emphasizes on manual and non-manual signals where the manual signs involve fingers, hands,
arms and non-manual signs involve face, head, eye or body.
Gesture Recognition has Wide Ranging Applications: Developing new standards for hearing
impaired, To detect the sign language, Communicating through video calls, To allow learning
capabilities to be shared from two or more different places through telephonic conversations.
The process of converting the signs and gestures shown by the user into text is called sign language
recognition. It bridges the communication gap between people who cannot speak and the general
public

Nigerian Sign Language, NSL is the national sign language of deaf people in Nigeria, however,
Nigeria does not have a national sign language yet. ASL (with a possible mix of Signed English)
was introduced in 1960, a few years after Ghanaian Sign Language, by Andrew Foster, a deaf
African-American missionary, thereby raising a signing system some scholars have referred to as
a dialect of ASL. Deaf education in Nigeria was based on oral method and existing indigenous
sign languages were generally regarded as gestural communication prior to Andrew Foster's
arrival. The conventional "Nigerian Sign Language" today has been described as the "School
Sign Language" and is coded as NSI on the repository of languages of the world by SIL
International. There is a Ghanaian influence in NSL; both are based on American Sign Language.
The School Sign Language has little relationship with the various Indigenous Nigerian sign
languages such as Hausa Sign Language, Yoruba Sign Language, and Bura Sign Language. The
Save the Deaf and Endangered Languages Initiative] and Nigerian National Association of the
Deaf have been working to document indigenous and national varieties of NSL both for research
and reference for the Nigerian deaf population.

IMAGE EXPRESSION DATABASE:

An image dataset was made from scratch. Different types of images were collected and altered
for each alphabet in sign language and were put under a single database and features of these
images were extracted which helped in recognition of sign language . In addition with these
alphabets some popular signs were also used so that the sign which is frequently used in daily
life could be included.

The advantage for this kind of database is that, this will be able to recognise the sign language of
different countries because the dataset would have all the variations of sign languages required,
otherwise all other system are limited to it’s own country sign language. All the images are
combined and collected into a single database to form a Collective database of Sign Language
used in the system.

1.3 Statement Of The Problem

Communication is one of the basic requirements for survival in society. Communication impaired
people communicate among themselves using sign language but normal people find it difficult to
understand their language.

Extensive work has been done on American sign language recognition and Nigerian sign
language does not differs significantly from American sign language but from Indian sign
language ( ISL) which uses two hands for communicating (20 out of 26) whereas ASL uses single
hand for communicating.

Using both hands often leads to obscurity of features due to overlapping of hands. In addition to
this, lack of datasets along with variance in sign language with locality has resulted in restrained
efforts in ISL gesture detection. Our project aims at taking the basic step in bridging the
communication gap between normal people and deaf and dumb people using American sign
language. Effective extension of this project to words and common expressions may not only
make the communication impaired people communicate faster and easier with outer world, but
also provide a boost in developing autonomous systems for understanding and aiding them.
The various advantages of building such a system includes:
● Sign-to-text translation system or dialog systems which are use in specific public
domains such as airports, post offices, or hospitals.
● SLR can help to translate the video to text or speech enables inter communication
between normal and deaf people.

1.4 Aim & Objectives of the study

Aim
 To build a Sign language recognition system using Convolutional Neural Network that
facilitates an Effective Communication between hearing impaired person and us across
different tribe.

Objectives

 To design a system to translate the sign language to its meaningful alphabet and show that
alphabet to a person with hearing impairment would help him/her and the people around
him/her in communicating easily..
 To evaluate the system to concentrates on a proper procedure for analyzing raw data
received from webcam using CNN algorithm. Database is created from the scratch to
resolve the shortcoming in other sign language system

1.5 Significance Of The Study

1.5.1 Problem Statement

 COMMUNICATION PROBLEM :

 Person with hearing impairment can’t understand their audience and not all of this
audience know their language.

 Two deaf people can’t communicate with each other unless both know the sign
language.
  REGIONAL SIGN LANGUAGE :

 Like our spoken language, tribes also have their own sign language. Again there could
be misunderstanding of sign language between two person if they belong to different
tribes

1.5.2 Solution
 In the proposed system the disabled or deaf person should provide a gesture or sign
image to the system. The system evaluates the sign input with OpenCV and classifies
the input to the recognized identification. The output will be shown in text format. This
is a prototype to develop the concept of converting sign language to text. The aim of
this project is to provide an application to the society to establish the ease of
communication between the deaf and mute people by making use of image processing
library.

1.6 Scope Of The Study

The project would be designed to enable communication impaired people communicate with
other people. The design will cover the sign language recognition and translation into text through
integration of the sign languages, python, Jupyter Notebook, Tensorflow Object Detection API,
OpenCV (Open Source Computer Vision Library), Label Image

1.7 Limitation of The Study

 CHAPTER 2
LITERATURE REVIEW

2.1 INTRODUCTION

The domain analysis that we have done for the project mainly involved understanding the neural
networks. Overview of sign language recognition system used for years

2.1.1 Python
Python is an interpreted, object-oriented, high-level programming language with dynamic
semantics. Its high-level built in data structures, combined with dynamic typing and
dynamic binding, make it very attractive for Rapid Application Development, as well as
for use as a scripting or glue language to connect existing components together. Python's
simple, easy to learn syntax emphasizes readability and therefore reduces the cost of
program maintenance. Python supports modules and packages, which encourages
program modularity and code reuse. The Python interpreter and the extensive standard
library are available in source or binary form without charge for all major platforms, and
can be freely distributed.

2.1.2 Jupyter Notebook

The Jupyter Notebook is an open-source web application that allows you to create and
share documents that contain live code, equations, visualizations and narrative text.
 It is a web-based interactive computational environment for creating Jupyter notebooks
documents. The "notebook" term can colloquially make reference to many different
entities, mainly the Jupyter web application, Jupyter python web server, or Jupyter
document format depending on context. A Jupyter Notebook document is a JSON
document,following a versioned schema, and containing an ordered list of input/output
cells which can contain code, text (using Markdown), mathematics, plots and rich media,
usually ending with the ".ipynb" extension.

2.1.3 Tensorflow Object Detection API

TensorFlow is an open-source software library for dataflow programming across a range

of tasks. It is a symbolic math library, and also used for machine learning applications
such as neural networks. It is used for both research and production at Google, often
replacing its closed-source predecessor, DistBelief. TensorFlow is Google Brain's second
generation system.

TensorFlow provides a Python API as well as C++, Haskell, Java, Go and Rust APIs.
Third party packages are available for C#,Julia, R, Scala and OCaml.
Among the applications for which TensorFlow is the foundation, are automated image
captioning software, such as DeepDream. RankBrain now handles a substantial number
of search queries, replacing and supplementing traditional static algorithm-based search
results.

2.1.4 Open CV
OpenCV (Open Source Computer Vision Library) is a library of programming functions
mainly aimed at real-time computer vision. It mainly focuses on image processing, video
capture and analysis including features like face detection and object detection.

2.1.5 Label-Image
 Labelling image is the first and most significant part of object detection. Image labelling
gives you insight into the content of images. When you use the API, you get a list of the
entities that were recognized: people, things, places, activities, and so on. Each label
found comes with a score that indicates the confidence the ML model has in its relevance.

2.1.6 Neural Networks

A neural network is a series of algorithms that endeavors to recognize underlying relationships
in a set of data through a process that mimics the way the human brain operates. In this sense,
neural networks refer to systems of neurons, either organic or artificial in nature. Neural
networks can adapt to changing input; so the network generates the best possible result
without needing to redesign the output criteria. The concept of neural networks, which
has its roots in artificial intelligence, is swiftly gaining popularity in the development of
trading systems.

A neural network works similarly to the human brain’s neural network. A “neuron” in a neural
network is a mathematical function that collects and classifies information according to a
specific architecture. The network bears a strong resemblance to statistical methods such as
curve fitting and regression analysis. Some areas of application of neural network includes:
Speech Recognition, Character Recognition, Signature Verification Application, Human Face
Recognition

2.2 EXISTING SYSTEM

In Literature survey we have gone through other similar works that are implemented in the domain of
sign language recognition. The summaries of each of the project works are mentioned below

(Sign-language recognition system to aid deaf-dumb People using Contourlet transform

Benjamin Joseph A, Mahesh S, Vignesh D, Prasannabalaji D, Sheshathri G, 2017)

Communication between deaf-mute and a normal person is a challenging job. The Sign Language
is a method to communicate with deaf-dumb people. The goal of the work is to design and
implement a real time Sign Language recognition system, to recognize 26 alphabets from Universal
English Sign Language using MATLAB . The proposed scheme consists of five modules: image
acquisition, pre-processing and segmentation, feature extraction, sign recognition and voice
conversion. The Database is created and developed, for training and testing phases. The hand signs
of the alphabets are acquired, transformed using Contourlet Transform and features were extracted,
then they are classified and recognized using KNN classifier and are stored in a file, finally the file
content is converted into audio using the external - software program. This work enables deaf-mute
people to communicate with non-signing people without the demand of an interpreter.
The aim of this task is to prepare an intelligent system which can work as a translator between the
sign language and the spoken language dynamically, and can get the communication between
people with hearing impairment and normal people both effective and effective. The signals are
stated in terms of voice and text for showing.

DATA ACQUISITION:

Capturing of the sign languages using a resolution of 380 X 420 pixel, Runtime images were
captured with web cameral under white background to avoid illumination effect. Image are also
captured using a specific distance of 1.5-2 ft between cameral and signer

IMAGE PREPROCESSING AND SEGMENTATION:

Preprocessing consist image acquisition, segmentation and morphological filtering methods.

Then the Segmentation of hands is carried out to separate object and the background. Otsu
algorithm is used for segmentation purpose. The segmented hand image is represented certain

features. These features are further used for gesture recognition Morphological filtering techniques
are used to remove noises from images so that we can get a smooth contour

FEATURE EXTRACTION

Feature extraction is a method of reducing data dimensionality by encoding related information

in a compressed representation and removing less discriminative data. Feature extraction is vital to
gesture recognition performance.

SIGN RECOGNITION:

In Sign pattern recognition, the K-nearest neighbor algorithm (K-NN)[3] is a widely used classifier
for classifying objects based on closest training examples in the feature space. The k-nearest
neighbor algorithm is the simplest classifier of all machine learning algorithms.

This work enables deaf-mute people to communicate with non-signing people without the demand
of an interpreter.

(A Neural Network Classifier based American Sign Language Recognition System Gauri Nagavkar,
Kedar Potdar 2017)

The aim and objective of this paper is to create a communication system for speech impaired
people with the aid of sensory glove which interpret hand gestures to American Sign Language

This paper introduces a communication system for the deaf-mute which helps them translate sign
language into computer text using a hand-glove and neural network back-end. The hand-glove is
fitted with flex sensors to measure finger movements. The electrical signals generated by the
sensors are then processed and transmitted to a computer system with the use of an Arduino
microcontroller. The processed signals are fed to a neural network trained to classify 26 letters of
the alphabet. The system comprises of a hand glove fitted with flex sensors connected to an
Arduino Uno microcontroller to detect finger movement as electrical signals. These signals are then
parsed by a neural network running on a back-end computer to recognize the corresponding letter
in ASL.

To use a neural network for real-time classification tasks, it first needs to be trained on a labelled
dataset and then deployed to perform classification based on live inputs. This section describes the
procedure used to build a dataset, pre-processing it and training the network for classification.

A neural network based classifier is developed to translate gestures in the ASL to text on a
computer system, which can be further used for different purposes serving as another input
mechanism for the deaf and dumb

(Conversion of Sign Language into Text Mahesh Kumar N B 2018)

Sign Language Recognition is one of the most growing fields of research area. Many new
techniques have been developed recently in this area. The Sign Language is mainly used for
communication of deaf-dumb people. This paper shows the sign language recognizing of 26 hand
gestures in Indian sign language using MATLAB. The proposed system contains four modules such
as: pre-processing and hand segmentation, feature extraction, sign recognition and sign to text. By
using image processing the segmentation can be done. Some of the features are extracted such as
Eigen values and Eigen vectors which are used in recognition. The Linear Discriminant Analysis
(LDA) algorithm was used for gesture recognition and recognized gesture is converted into text and
voice format. The proposed system helps to dimensionality reduction.

This system provides an opportunity for a deaf-dumb people to communicate with non-signing
people without the need of an interpreter.

Sign language recognition is an important application of gesture recognition. Sign language

recognition has two different approaches.
- Glove based approaches
- Vision based approaches.

Glove based approaches

It requires signers to wear a sensor glove or a colored glove. The task will be simplified during
segmentation process by wearing glove. The drawback of this approach is that the signer has to
wear the sensor hardware along with the glove during the operation of the system.

Vision based approaches

Image processing algorithms are used in Vision based technique to detect and track hand signs and
facial expressions of the signer. However, there are accuracy problems related to image processing
algorithms and these problems are yet to be modified.
There are again two different approaches in vision based sign language recognition:
-3D model based
-Appearance based
3D model based methods make use of 3D information of key elements of the body parts. Using this
information, several important parameters, like palm position, joint angles etc., can be obtained.
This approach uses volumetric or skeletal models, or a combination of the two. Volumetric method
is better suited for computer animation industry and computer vision. This approach is very
computational intensive and also, systems for live analysis are still to be developed.

Currently, this research works have focused mainly on the recognition of static signs of ISL from
images or video sequences that have been recorded under controlled conditions. By using LDA
algorithm for sign recognition operation the dimensionality will be reduced. Due to dimensionality
reduction the noise will be reduced and with high accuracy. In future this project will be enhanced
by determining the numbers which will be shown in words.

Recognition of static signs of ISL from images or video sequences that have been recorded under
controlled conditions.

(Static Sign Language Recognition Using Deep Learning, Lean Karlo S. Tolentino, Ronnie O.
Serfa Juan, August C, Thio-ac, Maria Abigail B, Pamahoy, Joni Rose R. Forteza, Xavier Jet
O. Garcia 2019)

A system was developed that will serve as a learning tool for starters in sign language that involves
hand detection. This system is based on a skin-color modeling technique, i.e., explicit skin-color
space thresholding. The

skin-color range is predetermined that will extract pixels (hand) from non–pixels (background).
The images were fed into the model called the Convolutional Neural Network (CNN) for
classification of images. Keras was used for training of images.

Provided with proper lighting condition and a uniform background, the system acquired an average
testing accuracy of 93.67%, of which 90.04% was attributed to ASL alphabet recognition, 93.44%
for number recognition and 97.52% for static word recognition, thus surpassing that of other related
studies. The approach is used for fast computation and is done in real time.

Index Terms—ASL alphabet recognition, sign language

recognition, static gesture. to develop a system that will recognize static sign gestures and convert
them into corresponding words. A vision-based approach using a web camera is introduced to
obtain the data from the signer and can be used offline. The purpose of creating the system is that it
will serve as the learning tool for those who want to know more about the basics of sign language
such as alphabets, numbers, and common static signs. The scope of the study includes basic static
signs, numbers and ASL alphabets (A–Z). One of the main features of this study is the ability of the
system to create words by fingerspelling without the use of sensors and other external technologies.

The system will be implemented through a desktop with a 1080P Full-HD web camera. The camera
will capture the images of the hands that will be fed in the system. Note that the signer will adjust
to the size of the frame so that the system will be able to capture the orientation of the signer’s
hand.
When the camera has already captured the gesture from the user, the system classifies the test
sample and compares it in the stored gestures in a dictionary, and the corresponding output is
displayed on the screen for the user.

To develop a system that can translate static sign language into its corresponding word equivalent
that includes letters, numbers, and basic static signs to familiarize the users with the fundamentals
of sign language. The researchers created an assessment plan, and a series of tests were done to
guarantee the significance of the functionalities of the system intended for non-signers.

Results of testing were remarkable marks in terms ofusability and learning impact of the system.
This project was done with proper aid and consultation from sign language experts.

The system only recognized five Filipino words and usedcolored gloves for hand position
recognition, thus having the best accuracy hence, the system can be helpful in learning static sign
language.

(Classification of Sign Language Gestures using Machine Learning, Abhiruchi Bhattacharya Vidya
Zope, Kasturi Kumbhar, Padmaja Borwankar, Ariscia Mendes 2019)

The low awareness of sign language among the general public presents a hurdle in communication
with the deaf and dumb communities and their integration within society. A sign language
translator application can help reduce this communication gap.
This paper describes the development of an application that aims to detect and differentiate finger spelling
gestures from video frames using image processing and machine learning techniques, and discusses results
and future prospects.

This paper presents a system developed to detect finger spelling gestures from video and give their
English equivalent letter using machine learning, with a focus towards developing a potential
solution for everyday use. A classifier is trained on a dataset of 24 finger spelling gestures using the
bag of visual words approach, wherein features detected in the images are clustered to form a
codebook, then each image is expressed as a histogram denoting frequency of observed code words
in that image.

In the presently developed system, first a machine learning classifier is trained to differentiate
between images of static finger spelling gestures. The trained classifier is then used to detect finger
spelling signs from video and give their corresponding English letter. The computer system used
for development has the following specifications: Intel Core i5-7200U CPU @ 2.5 GHz and 8.00
GB RAM. We use OpenCV library for image and video processing and scikit-learn in Python for
training and testing classifiers

The system can satisfactorily recognize certain finger spelling signs with reasonable accuracy.
However, it could not distinguish all characters because of small dataset and errors in skin
segmentation. It can be concluded from the results discussed above that for classification using
video, the performance depends upon lighting and skin masking accuracy. Even with well-trained
models, these become crucial factors in determining the overall accuracy of the system.
Detect finger spelling gestures from video and give their English equivalent letter using machine
learning,

(Recognition of American Sign Language using Image Processing and Machine Learning
Sharmila Gaikwad, Akanksha Shetty, Akshaya Satam, Mihir Rathod, Pooja Shah 2019)

In this era of touch screen devices, human machine interaction is a very important factor nowadays.
Many devices are developed which can be operated without touching the system. So, in this project
we are going to demonstrate how one can operate the system without actually touching the system
but with the use of gestures.

in this project we are going to demonstrate how one can operate the system without actually
touching the system but with the use of gestures. Human–Computer Interaction (HCI) can be
described as the point of communication between the human user and a computer.

Image Acquisition
The initial phase of Image Acquisition is obtaining a picture during runtime via integrated camera
and while obtaining these pictures will be loaded in the directory after they are captured.

Feature Extraction
For any of the object, there are many features, interesting points on the object, which can be
extracted to provide a “feature” description of the object.

Orientation Detection
This step will take the input of hand movement in any of the form and the gesture will be identified
by the described part of feature extraction since the SIFT algorithm includes the orientation
assignment procedure.

Gesture Recognition
On the completion of the entire process the application will be then translated into its recognized
character or alphabet from the gesture which is beneficial to be understood in layman‟s language.

Text to Speech
When the character is get selected based on recognized sign using speech conversion, recognized
text is converted to speech and an audio output is executed

This paper principally dedicated to the ASL system. The system will provide an interface that can
easily communicate with deaf people by Sign Language Recognition. The system is not only can
apply in family environment, but also can apply in public. For the Social use this system is very
helpful for deaf and dumb people. We will build simple gesture recognizer based on OpenCV
toolkit and integrated it into Visionary framework. The project involves distinguishing among
different alphabets of English language. Also, the project is included with recognition of all the
English alphabets & numbers. Further, we may move on to recognizing words, from as large a
dictionary as possible, from Indian Sign Language.

(Tamil Sign Language Recognition using Machine Learning to Aid Deaf and Dumb People SJ.
Swetha1,Dr. S. Roselin Mary 2019)

Sign language recognition is an emerging research field to aid deaf and dumb people. The research
on recognition and bringing best results is challenging in this area of interest. The overcome the
challenges, the proposed system uses convolutional neural networks for training the dataset images
of various sign with output in sound wave files, which will be effective solution for both deaf and
dumb. This paper proposes a method that provides the conversion of a set of few combinations of
the signs, which represents positions of fingers and palms into Tamil letters.

This paper deals with a system which recognizes the Tamil Sign Language for human– computer
interaction. To overcome the challenges in sign language system and to make more useful for deaf
and dumb people who knows Tamil as only known language.

To propose sign language system using convolutional neural networks for training the dataset
images of various sign with output in sound wave files. For the conversion of signs, image
processing and feature extraction method is used, training phase is done using Conv2D and testing
phase is done. The static images and dynamic images (from web camera) can be converted by these
methods.

The hand sign recognition algorithm is proposed. The hand sign of Tamil alphabets are acquired
using a digital camera. The still images of these hand signs are in RGB color space and it is
converted to HSV color model for foreground extraction. The image is converted to gray scale
images. Deep learning techniques of convolutional neural network from keras is applied for
classification. In the training phase, the signature of hand signs are stored as images along with the
corresponding voice wave files. In the testing phase, a person shows the hand signs of Tamil
alphabets as an input and the features are extracted. The features of training hand signs set are
extracted and stored in trained model for template matching.

This proposed method is very useful for the deaf and dumb people to communicate with normal
people in Tamil letters. This approach can handle different types of alphabets in a uniform
background. In future the dynamic gestures can be considered for creating words or sentences.

(Sign Language and Gesture Recognition for Deaf and Dumb People Hema B N, Sania
Anjum, Umme Hani, Vanaja P, Akshatha M 2019 )
Every day we see many people who are facing illness like deaf, dumb and blind etc. They face
difficulty to interact with others. Previously developed techniques are all sensors based and they
didn’t give the general solution. This paper explains a new technique of virtual talking without
sensors. An image processing technique called Histogram of gradient (HOG) along with artificial
neural network (ANN) has been used to train the System.

Aim of our project is to develop a concept of virtual talking system without sensor for people who
in need, this concept achieving a by using image processing and human hand gesture input. This
mainly helps to people who can’t talk with other people

In the proposed system the unable or dumb person should provide a gesture or sign image to the
system. The system evaluates the sign input with matlab image processing technique and classifies
the input to the recognized identification. Later it initiates the voice media through the system when
the input image matches with the given dataset. And the output will be shown in the text format
too. This is a prototype to develop the concept of converting the sign language to speech and text.
The aim of this paper is to provide an application to the society to establish the ease of
communication between the deaf and mute people by making use of image processing algorithm

The sensor less sign language and gesture recognition system is a module which provides an easy
and satisfactory user communication for deaf and dumb people. The module provides two way
communications which helps in easy interaction between the normal people and disables. The
system is novel approach to ease the difficulty in communicating with those having speech and
vocal disabilities. The aim is to provide an application to the society to establish the ease of
communication between the deaf and mute people by making use of image processing algorithm.
The module provides two way communications which helps in easy interaction between the normal
people and disables

(Sign Language Converter For Deaf And Dumb People, Shweta S. Patil, Mr .G.P.Jain 2019)

Deaf and dumb people learn sign language for communication between deaf-mute and normal
people. But normal people are unable to understand what the dumb and deaf people are trying to
say. Normal people do not have time to learn the sign language. It causes barrier in communication
between deaf-dumb and normal people. The work presented in this paper mainly reduces the
communication gap between deaf-dumb and ordinary people.

This paper aims to convert the hand gestures based on electronic device that translate sign language
into text and speech to make the communication between the deaf-mute communities with the
general public.

This paper use the hand gesture recognition techniques

1 Glove Based Techniques

2 Vision Based Techniques

3. Hybrid Techniques
The communication between a deaf and hearing person causes a serious problem compared to
communication between blind and normal visual people. The review paper aims to facilitate people
by means of a embedded based or image processing based communication interpreter system. Deaf
people require interpreter system & Hearing machine. Dumb people require interpreter System
only.

(Sign language Recognition Using Machine Learning Algorithm, Prof. Radha S. Shirbhate,
Mr. Vedant D. Shinde, Ms. Sanam A. Metkari, Ms. Pooja U. Borkar, Ms. Mayuri A. Khandge
2020 )

An Sign Language is one of the way to communicate with deaf people. In this work sets, included
features and variation in the language with locality have been the major barriers which has led to
little research being done in ISL. One should learn sign language to interact with them.

Learning usually takes place in peer groups. There are very few study materials available for sign
learning. Because of this, the process of learning sign language learning is a very difficult task

This project aims at identifying alphabets in Indian Sign Language from the corresponding
gestures. Gesture recognition and sign language recognition has been a well researched topic

for American Sign Language(ASL), but few research works have been published regarding Indian
Sign Language(ISL). But instead of using high-end technology like gloves or kinect, we aim to
solve this problem using state of the art computer vision and machine learning algorithms.

On collected dataset, we divided our approach to tackle the classification problem into three stages.
The first stage is to segment the skin part from the image, as the remaining part can be regarded as
noise w.r.t the character classification problem. The second stage is to extract relevant features
from the skin segmented images which can prove significant for the next stage i.e learning and
classification. The third stage as mentioned above is to use the extracted features as input into
various supervised learning models for training and then finally use the trained models for
classification.

In this work, we have gone through an automatic sign language gesture recognition system in real-
time, using different tools. Although our proposed work expected to recognized the sign language
and convert it into the text, there’s still a lot of scope for possible future work.

Recognized the sign language and convert it into the text Using state of the art computer vision and
machine learning algorithms. collecting a dataset and then use various feature extraction techniques
to extract useful information which is then input into various supervised learning techniques.

(Sign Language Recognition Using Neural Network Shailesh Bachani, Shubham Dixit, Rohin
Chadha, Prof. Avinash Bagul 2020)

Hand Gesture recognition system provides us an innovative, natural, user friendly way of
interaction with the computer which is more familiar to human beings. Gesture Recognition has a
wide area of application including human machine interaction, sign language, immersive game
technology etc. By keeping in mind the similarities of human hand shape with four fingers and one
thumb,

This paper aims to present a real time system for hand gesture recognition on the basis of detection
of some meaningful shape based features like orientation, center of mass (centroid), status of
fingers, thumb in terms of raised or folded fingers of hand and their respective location in image.
The system can capture hand gestures and navigate the words as in text format it will use for muted
persons to see these words and understand the sentences.

● It is also usable for blind peoples because they need to communicate with normal persons (using
speech recognition).

The system converts the Gestures video into simple words in English as well as make a sentence of
that each word in English. The CNN process used in the video processing module gives the
matched results. Based on the right match, the Sign Writing Image File is retrieved and stored in a
folder. This folder served as the input to the Natural Language Generation Module.

Developed system for muted persons which they can interact with normal persons so the American
Sign Language System (ASL) introduce to manage the problem, the system applying neural
networks classification approach for identifying the hand gestures like alphabets or numbers so the
system can predict accurate result of hand sign and make the words as in the form of sentences.

The system converts the Gestures video into simple words in English as well as make a sentence of
that each word in English. The system converts the Gestures video into simple words in English as
well as make a sentence of that each word in English.

(Sign Language Recognition System For People With Disability Using Machine Learning And
Image Processing, Bayan Mohammed Saleh, Reem Ibrahim Al-Beshr, Muhammad Usman
Tariq 2020)

In a tech era, no one Should be limited due to his or her inability. The application of technology
should create a platform or a world of equality

Despite the natural state of humans. On the other hand, technology is the most innovative thing on
Earth for every time the clock ticks, researchers, software engineers, programmers, and information
technology specialists are always coming up with bright ideas to provide Convenience to everyone.
This paper shows how artificial intelligence is being used to help people who are unable to do what
most people do in their everyday lives.

The paper objective is to blend deaf and dumb within society and make them able to use their
personal computers more effectively and efficiently. Our idea is to create sign assistance, like many
applications which is using voice assistance such as Siri on iOS and Cortana on windows. There is
need to develop an application that will create an interactive platform where the sign language can
be translated to voice output and writing, and voice and writing input can also be converted to sign
language
This paper presents an understanding of complex hand movements. A framework based on Hidden
Markov Models (HMMs) [11] is provided for modeling and recognition of complex gesture
trajectories. A detection algorithm is used to detect the hand of the user, and a contour-based hand
tracker is developed, which combines condensation and partitioned sampling. The proposed
approach can attain automated online identification of hand movements and can effectively reject
atypical movements. The experimental results indicate

that the proposed algorithm can produce better results for recognition than the conventional method
of hand recognition [12]. The hand gesture recognition system consists of three major parts: palm
detection, hand tracking, and trajectory recognition. Figure 5 provides an overview of the hand
gesture recognition process. The hand tracking function is enabled when the device senses an open
hand in front of the camera; when the user finishes the gesture, the hand gesture classification
based on HMM is disabled. The basic algorithmic structure for our process of recognition is the
following

1. Detect the palm from the video and initialize the tracker with the template of hand shape.
2 Track the hand motion using a contour-based tracker and record the trajectory of the palm
center.
3 Classify the gesture using HMM, which gives the maximum probability of occurrence of
observation Sequence.

This paper development will serve to bring efficiency in communication, not only to the deaf and
dumb but those with the ability to hear and speak as well. In addition to creating opportunities for
their career growth, it will enhance their social life through effective communication.

Making an impact and changing the lives of the deaf and dump through technology will be an
innovation of the year worth the time and resources.

This system provides detailed hand gestures that show the interpretation at the bottom so that
everyone can understand them.

(Hand Gesture Recognition System for Deaf and Dumb Syed Raquib Shareef, Mohammed
Mannan Hussain, Akash Gupta, Akeem Aejaz Aslam 2020)

The problem existing at the moment is that most of the people are not able to comprehend hand
gestures or convert them to the spoken language quickly enough for the listener to understand. In
addition to this communication with sign language is not a very easy task. This problem demands a
better solution which can assist speech impaired population converse without any difficulties.

In this paper, the authors develop an artificial mouth to ease the communication process of speech-
impaired people. We design one glove which is fitted with the flex sensor in our hand then the flex
sensor senses the signal and this signal given to the microcontroller whereas all the data kept in the
database, then microcontroller matches the motion of hand with the database and produces the
speech signal and text

The proposed system aims at reducing the communication barrier between the speech impaired and
normal people thus assisting them for effective communication and easing their difficulties. Our
prototype involves Raspberry pi, Arduino nano as an A/D converter which are interfaced with flex
sensors and accelerometer for reading the hand gestures, speaker module, OLED, hand glove and a
power supply of 5v. When a specific sign is made, the flex sensors gets bended and unique values
are generated. These values are stored in the database and when the sign is made the
microcontroller matches the values with the stored values and the output is made available in the
form of text on the OLED and a audio from the speaker which are connected to the raspberry pi.

We were able to develop an efficient gesture recognition system that did not utilize any markers
and camera hence making it more user and cost-friendly.

Efficient gesture recognition system with the aid of flex sensor

(Sign Language And Gesture Recognition For Deaf And Dumb People, Nagma Neha, Prof.
Aditi Ravichandra 2020)

Human-computer interaction is very much essential part of most people's daily life. The human and
computer interaction mode is through keyboard, mouse, joystick, and wireless input devices.
Interaction between people and computers have made it easier for people to operate the computer
and improve work efficiency these days. But what for those who actually cannot be a part of the
computer technology in these days.

Aim for this research is to develop an algorithm that will translate the ISL into English, so that even
Deaf and Dumb people can learn English. The accuracy of this system is as high as 97.5% and can
help dumb and deaf people and can also reduce illiteracy percentage in INDIA.

The Implementation includes the following steps:

1. Image Segmentation
2. Orientation Detection
3. Features Extraction
4. Classification of bits Generation
5. Hand Gesture interpretation

In our system, for producing the interaction between gesture recognition software and MS
Office/notepad we have used the Abstract Windows Toolkit. It is used to control the mouse and
keyboard remotely. For providing the human computer interaction through gesture recognition
system we have generated some of the key press events as shown in Table-3 Below. The sample
hand gestures get converted into corresponding key press events which we have assigned.

Hand gesture recognition can be done by wearing gloves this proposed system can work for real-
time translation of Taiwanese sign language.

(Real-Time Recognition of Sign Language Using Machine Learning, Shubham Sawant,

Ronak Sahay Prathamesh Salunkhe Prof. Roshan bauskar 2021)
Sign Language may be a language within which we tend to create use of hand movements and
gestures to communicate with other people who are chiefly deaf and dumb.

In our project we basically focus on producing a model which can recognize in fingerspelling based
hand gestures in order to form a complete word by combining each gesture.

This paper proposes a system

to recognize the hand gestures employing a Deep Learning Algorithm, Convolution Neural
Network (CNN) to map the image and predict the gestures. This paper shows the sign language
recognition of 26 alphabets and 0-9 digits hand gestures of yank linguistic communication. The
planned system contains modules like pre-processing and Our method provides 95.7 % accuracy
for the 26 letters of the alphabet extraction, coaching and testing of model and sign to text
conversion. In this project we have used ML, OpenCV and Tensor Flow to recognize face masks.
Our dataset got a higher accuracy in recognition. Our method provides 95.7 % accuracy for the 26
letters of the alphabet

In this report, a functional real time vision based American sign language recognition for D&M
people have been developed for asl alphabets.

This paper uses python based system to predict hand gesture using the loaded dataset

2.3 INFERENCES DRAWN OUT OF LITERATURE SURVEY

Since there didn’t exist any such system which could detect Nigerian sign language which can be
used across the 36 states in Nigeria, we developed a system that could detect Nigerian Sign
Langauge. Thus, creating a single platform for all hearing impared in Nigeria. Now a person with
speaking disability would not have to switch between different app or would have to learn
different sign languages when trying to communicate with someone who is not a native.
Learning and getting fluent in only one language will enable the disabled persons to
communicate confidently and more fluently and would remove any sort of barrier thus freeing
their path of this obstacle.
CHAPTER 3

Methodology