Professional Documents
Culture Documents
using the sign language. The main problem with this kind of communication is that
the non-impaired people, who cannot understand the sign language, would not be able
to communicate with these people or vice versa. The project is intentionally designed
to allow deaf and dumb communities to convey messages and connect with the
society. It aims to bridge the gap between speech and hearing impaired people and the
ability to communicate effectively. Many existing studies have proposed methods for
sign language recognition The project uses a machine learning system to identify,
especially English alphabetic sign language, and convert it into text. The focus of this
work is to create a vision based application that offers American Sign Language
translation to text thus aiding communication between signers and non-signers. The
proposed model takes image sequences and extracts temporal and spatial features
from them.
CHAPTER 1
INTRODUCTION
1.1 BACKGROUND
The sign language is used widely by people who are deaf-dumb these are used as a
medium for communication. A sign language is nothing but composed of various
gestures formed by different shapes of hand, its movements, orientations as well as
the facial expressions. There are around 466 million people worldwide with hearing
loss and 34 million of these are children. `Deaf' people have very little or no hearing
ability .They use sign language for communication. People use different sign
languages in different parts of the world. Compared to spoken languages they are very
less in number. India has its own sign language by the name Indian Sign Language
(ISL). In developing countries there are only very few schools for deaf students.
Unemployment rate among adults with hearing loss are very high in developing
countries . Data from Ethnologue states that among deaf population in India, which is
about 1% of total population, literacy rate and number of children attending school is
very less. It goes on to state that official recognition of sign languages, increasing the
availability of interpreters and providing transcription in sign languages greatly
improve accessibility. Signs in sign languages are the equivalent of words in spoken
languages Signed languages appear to favor.
The associate editor coordinating the review of this manuscript and approving it for
publication was Weiyao Lin. Simultaneous sign internal modification, rather than the
concatenation of morphemes. But learners in the initial stages of SL learning use
iconicity as a mnemonic aid to remember new signs. Butthe lack of iconicity makes it
difficult to learn new signs for those who learn SL as a new language. Finger spelling
is the representation of the letters of a writing system and sometimes numeral
systems. Sign Language (ISL) can represent English alphabets A-Z using finger
spelling. It can be one handed or two handed andi ISL follows two handed style. It
issued to represent words that have no sign equivalent or used ito emphasize a word
Though finger spelling usage is less [7], [8] in casual signing, they are an important
component in sign i language learning. This project aims at identifying alphabets in
Indian Sign Language from the corresponding gestures. Gesture recognition and sign
language recognition has been a well researched topic for American Sign
Language(ASL), but few research works have been published regarding Indian Sign
Language(ISL). But instead of using high-end technology like gloves or kinect,
we aim to solve this problem using state of the art computer vision and machine
learning algorithms.
1.2 RELEVANCE
The main aim of this project is to recognize the Sign language Recognition Using
Machine Learning Algorithm
1.5 SUMMARY
This chapters gives introduction about ISL and used different techniques, from
that techniques we have used one of technique for sign language.
CHAPTER 2
LITERATURE SURVEY
Paper [1] demonstrates, a hand free demonstration of Taiwanese data language which
uses the wireless system to process the data. To differentiate hand motion, they have
inner sensors put into gloves to show the parameters as given by, posture, orientation,
motion, defined of the hand in Taiwanese Sign Language could be recognize in no
error. The hand gesture is considered by flex inner sensor and the palm size
considered using the g sensor and the movement is considered using the gyroscope.
Input signals would have to be consider for testing for the sign to be legal or not
periodically. As the signal which was sampled can stay longer than the pre-set time,
the legal gesture sent using phone via connectivity like Bluetooth for differentiating
gestures and translates it. With the proposed architecture and algorithm, the accuracy
for gesture recognition is quite satisfactory. As demonstrated the result get the
accuracy of 94% with the concurrent architecture.
Having a real-time sign language detector increases the efficiency of the community
to able to in contact with people having disabilities like hearing and deaf society.
Authors [2] have using machine learning algorithms presented the idea of a translation
with skin colour tone to detect the ASL. They have made a skin colour segmentation
that automatically depicts the colour and give the tune to it for further detection. They
have used YCbCr spacing of colour as its vastly used in video template code and
gives the efficient results of colour tone for human skin. Further they have taken the
CbCr plane to distribute the skin tone colour. People from Different ethnicity have
their tones different which is crafted in a model. Deep Learning methods and several
machine learning algorithms are used to demonstrate translator to translate between
parties.
Authors of paper [3] applied a method of using a synthetic named animation making
approach they have converted Malayalam language to Indian sign language. The
intermediate representation in this method for sign language is being used by
HamNoSys. In this method the application accepts some sets of words, say, either one
or more and forms It in animated portion. Theres an interactive system which further
converts the portion of words in to HamNoSys designed structure. Its application pars
everything that has been designed as it used by Kerala government to teach about sign
language and subtle awareness.
Having to communicate between deaf people and normal public has become a
difficult task now days and to implement a such as the society lacks a good translator
for it and having an app for it in our mobile phones is like having a dream at day.
Authors [4] have proposed something great for the deaf community or hearing aid
community by providing an app for the communication. But making an app for it is
no simple task at it requires lot of efforts like memory utilization and a perfectly fined
design to implement a such. What their application does is that they take a pictureof a
sign gesture and later converts is to a meaningful word. At first, they have compared
the gesture using histogram that has been related to the sample test and moreover
samples that are obliged to BRIEF to basically reduce the weight on the CPU and its
time. They have explained a process on which on their app, its very easy to add up a
gesture and store it in their database for further and expand detection set. So lastly,
they came strong with having an app as a translator instead of several applications that
are being used lately by users.
This paper [5] is completely based on a Spanish speaking language which converts the
basic words into the Spanish language which is good for Spanish deaf people as it will
provide them a stance to understand the sign language at pace as itll be converted in a
Spanish language rather than in English which is popularly used as ASL. The device
or an application that they have made for this comprises of many terms such as visible
interface which is used by the deaf person to specify the sequence of sign data, a
translator, which simply converts those series in Spanish based language in a formed
series, and convertor to speech, which basically converts those entire bits into a
meaningful sentence in Spanish, of course. They mainly focus on the interface which
is designed visually for the impaired which has shown many ways to writing the sign
language in a real time.
The authors [6] have built a system which works in a continuous manner in which the
sign language gesture series is provided to make a automate training set and providing
the spots sign from the set from training. They have proposed a system with instance
learning as density matrix algorithm that supervises the sentence and figures out the
compound sign gesture related to it with a supervision of noisy texts. The set at first
that they had used to show the continuous data stream of words is further taken as a
training set for recognizing the gesture posture. They have experimented this set on a
confined set of automated data that is used for training of them, identification for them
and detection stored a subtle sign data to them. It has been stored around thirty sign
language data that was extracted from the designed proposal.
Further the authors [7] explained, the lack of automated structure to translate
symptoms from LSM makes integration of listening to-impaired human beings to
society extra difficult. This painting affords a totally new technique for LSM
alphanumerical signs popularity based on 3D Haar- like features extracted from depth
pictures captured by means of the Microsoft Kinect sensor. Features are processed
with a boosting set of policies. To observe normal performance of our technique, we
Identified a tough and speedy sign from letters and numbers, and in comparison, the
results with the use of traditional 2D Haar-like capabilities. Our gadget is capable of
recognize static LSM signs and signs with a higher accuracy percentage than the one
obtained with extensively used 2D features.
As It has been preferred for a society that having a sign language for hearing impaired
and deaf people to communicate. As coming of several technologies in the
past, its been kind of easy to have a translator which converts the sign language to the
appropriate sentence and quite popular too. As they have shown this in their paper [8],
its firstly based on an Arabic sign language which automates the process of being
translated on to give a subtle way of communication and further they have shown that
the scope of their project apars the usage and defined set of measurements. The
application directly converts the Arabic sign language into a meaningful sentence by
applying an automated Machine learning algorithm as they concluded.
CHAPTER 3
Pre-processing
Pre-processing
Feature Machine
Feature
Extraction learning
Extraction
Module
Audio/Text
OUTPUT
Fig : Block Diagram of Indian Sign Language
Feature Extraction
As describing our own features may not result in higher efficiency, we started with
features as it computes the key points in the image which is more apt than describing
features manually. So, after the skin segmented images were obtained using the
model, we used the following approaches for extracting feature vectors.
Image Preprocessing
Image preprocessing contains cropping, filtering, brightness &contrast adjustment &
many more. To do such process Image enhancement, Image cropping & Image
Segmentation methods are used. Captured Images are in the form of RGB. So the first
step is to convert RGB images to binary images then cropping of image is to be done
so that unwanted part of i images can be removed. And now enhancement can be done
in certain selected area. In Image segmentation, Edge detection method is used which
can detect the boundary of cropped images which is further used for feature extraction
method.
The above table shows the number of points a gesture is matching with the features of
stored database. For example, the first colunm gesture which is showed is Indian Sign
language 'A', and it's clearly observed that there are 284 points matched with A
database. If any alphabet is given as input then the corresponding alphabet would
have highest number of matching point which enables us to display the equivalent text
result.
Random Forest
Since this was a 26 class problem, we tried our luck at Random Forest with HOG
feature vectors on the compressed images. It fell a little short of the Multiclass SVM
with 46.45% 4 fold CV accuracy. Our interpretation is that implementation of
decision trees in python is monothetic which may have led to formation of rectangular
regions whereas the actual boundaries may not have been rectangular.
Hierarchical Classification
One of the major reasons for the comparatively lower accuracy is the large number of
classes(26). So one approach we tried was breaking the classification problem into
multiple levels or hierarchical classification . First we train a linear kernel SVM
model to classify alphabets as one handed or two handed. This model performed with
95% accuracy. Then we trained linear kernel Multiclass SVM models to classify the
one handed alphabets(56% accuracy) and two handed alphabets(60% accuracy) and
then put the system together. An alphabet is first classified as one handed or two
handed, and then depending on the classification, is put into the corresponding model
and given a label. Even though the individual models performed better than the direct
multiclass SVM on HOG features, overall the performance was nearly the same and
four fold CV accuracy of 53.23% was observed.
ALGORITHM:
SVM is a Support Vector Machine:
According to Wikipedia, SVM is a supervised machine learning model with
associated learning analysis data.
In this given a set of training example, we divide data into two classes on the basis
of its labeling. If data is labeled it is put in category of supervised else in the category
of unsupervised.
When data is labeled then supervised SVM can be used, else SVM is not possible.
In case of unsupervised data SVM clustering algorithm is used. Uses of SVM:
They are used in text and hypertext classification.
SVM is used in hand written characters Recognization.
They are used in image classification.
Random Forest:
Random Forest is a popular machine learning algorithm that belongs to the supervised
learning technique. It can be used for both Classification and Regression problems in
ML. It is based on the concept of ensemble learning, which is a process of combining
multiple classifiers to solve a complex problem and to improve the performance of the
model.
Since this was a 26 class problem, we tried our luck at Random Forest with HOG
feature vectors on the compressed images. It fell a little short of the Multiclass SVM
with46.45% 4 fold CV accuracy. Our in perpetration is that implementation of
decision trees in python is monothetic which may have led to formation of rectangular
regions whereas the actual boundaries may not have been rectangular.
Logistic Regression:
Logistic regression is very popular in machine learning and statistics. It can work
on both binary and multiclass classification very well. I wrote tutorials on both
binary and multiclass classification with logistic regression before. This article will be
focused on image classification with logistic regression.
o Logistic regression is one of the most popular Machine Learning algorithms,
which comes under the Supervised Learning technique. It is used for
predicting the categorical dependent variable using a given set of independent
variables.
o Logistic regression predicts the output of a categorical dependent variable.
Therefore the outcome must be a categorical or discrete value. It can be either
Yes or No, 0 or 1, true or False, etc. but instead of giving the exact value as 0
and 1, it gives the probabilistic values which lie between 0 and 1.
o Logistic Regression is much similar to the Linear Regression except that how
they are used. Linear Regression is used for solving Regression problems,
whereas Logistic regression is used for solving the classification problems.
o In Logistic regression, instead of fitting a regression line, we fit an "S" shaped
logistic function, which predicts two maximum values (0 or 1).
o The curve from the logistic function indicates the likelihood of something such
as whether the cells are cancerous or not, a mouse is obese or not based on its
weight, etc.
o Logistic Regression is a significant machine learning algorithm because it has
the ability to provide probabilities and classify new data using continuous and
discrete datasets.
IMPLEMENTATION
Open source Computer Vision (OpenCV) is an image processing and computer vision
library mainly developed for artificial vision. It has a BSD license (free for
commercial or research use). OpenCV was originaly written in C, but currently, it's a
whole C++ interface, and there's additionally an entire Python interface to the library.
Open-source computer Vision Library, also called OpenCV, is associated with a
freeware software package aimed toward computer vision. It is used in this project
because of its versatility and the fact that it has a C++ interface. OpenCV runs on
most major Operating Systems (OS), making it worthwhile to use another computer to
program or test.
EXPERIMENTATION
CHAPTER 6
The preprocessed data is passed through several classifiers (Random Forests, Support
Vector Machines, Logistic Regression, Multilayer Perceptron) to draw effective
results. The accuracy of the new models has been found significantly higher than the
existing model.
Various types of techniques can be used to implement the classification and
recognition of images using machine learning. Apart from recognizing static images,
work has been done in the field of depth-camera sensing and video processing. A
diversity of processes embedded in the system was developed using various other
programming languages to implement the procedural techniques for the final system's
maximum efficacy. The problem can be solved and systematically organized into
three similar approaches which are, firstly using static image recognition techniques
and preprocessing procedures.
Output:- Trained model will recognize which character is there and then it will
convert text to speech using this speaker will recognize which character is there.
CHAPTER 7
CONCLUSION
In this work, we have gone through an automatic sign language gesture recognition
system in real-time, using different tools. Although our proposed work expected to
recognized the sign language and convert it into the text, there’s still a lot of scope for
possible future work.
Thus system was developed to translate the double handed Indian Sign Language into
both text and speech This system helps and aid the hearing impaired and mute people
to live independently. It develops confidence and will power to share their emotions,
thoughts, ideas and difficulties with the nominal people in the society. This eliminates
the gaps among the people and achieve better society This system is also useful to all
the people in the society gets as through this speech translation is possible, system
output can be obtained in different languages.
CHAPTER 8
REFERENCES
[I] MJerin Jose V.Priyadharshni et.al, "Indian Sign Language (ISL) Translation
Systern For Sign Language Learning,"IJIRD, vol. 2, May,2013, pp. 358-365.
[7] Shi, 1., and C. Tomasi, "Good Features to Track," Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, June 1994, pp. 593600
[II] A Park, S. Yun, 1. Kim, S. Min, and K Jung. "Real time vision-based Korean
finger spelling recognition system". Proc. World Acaderny of SET, 34:293298, 2008.
[14] Rekha, K; Latha, B. "Mobile Translation System from Speech Language to Hand
Motion Language", Intelligent Computing Applications (ICICA), 2014 International
Conference on, On page(s): 411-415.
[16] Och J., Ney H. "Improved Statistical Alignment Models". Proc. of the 38th
Annual Meeting of the Association for Computational Linguistics, pp. 440-447,
Hongkong, China, Octubre 2000.
[17] http://www.statmt.org/moses
[18] Vidal E., Casacuberta F, García P. “Gramatical Inferrence and Automatic Speech
Recognition”. New Advances and Trends in Speech Recognition and Coding (volume
147 of NATO-ASI Series F: Casacuberta and Vidal
[19] Papineni, K., Roukos, S., Ward, T., and Zhu, W. J. "BLEU: a method for
automatic evaluation of machine translation" in ACL-2002: 40th Annual meeting of
the Association for Computational Linguistics pp. 311-318.
[20] http://www.nist.gov/speech/tests/mt/
Chapter 3
SOFTWARE REQUIREMENTS
SPECIFICATIONS
and Characteristics
Login
Open CV
View Result
Mouse : Logitech.
MySQL :
MySQL, the most popular Open Source SQL database management system, is
developed, distributed, and supported by Oracle Corporation. The MySQL Web site
(http://www.mysql.com/) provides the latest information about MySQL software.
MySQL is database software which is used to store all database related activities
regarding to the project and it is easily stored and retrieve the data.
Technology Used:
PYTHON
Python is a high-level, interpreted, interactive and object-oriented scripting language.
Python is designed to be highly readable. It uses English keywords frequently where
as other languages use punctuation, and it has fewer syntactical constructions than
other languages.
Python is Interactive − You can actually sit at a Python prompt and interact
with the interpreter directly to write your programs.
History of Python
Python was developed by Guido van Rossum in the late eighties and early nineties at
the National Research Institute for Mathematics and Computer Science in the
Netherlands.
Python is derived from many other languages, including ABC, Modula-3, C, C++,
Algol-68, SmallTalk, and Unix shell and other scripting languages.
Python is copyrighted. Like Perl, Python source code is now available under the
GNU General Public License (GPL).
Python Features
Python's features include −
Easy-to-read − Python code is more clearly defined and visible to the eyes.
A broad standard library − Python's bulk of the library is very portable and
cross-platform compatible on UNIX, Windows, and Macintosh.
Interactive Mode − Python has support for an interactive mode which allows
interactive testing and debugging of snippets of code.
Portable − Python can run on a wide variety of hardware platforms and has
the same interface on all platforms.
Extendable − You can add low-level modules to the Python interpreter. These
modules enable programmers to add to or customize their tools to be more
efficient.
Scalable − Python provides a better structure and support for large programs
than shell scripting.
Apart from the above-mentioned features, Python has a big list of good features, few
are listed below −
It provides very high-level dynamic data types and supports dynamic type
checking.
It can be easily integrated with C, C++, COM, ActiveX, CORBA, and Java.
Win 9x/NT/2000
OS/2
PalmOS
Windows CE
Acorn/RISC OS
BeOS
Amiga
VMS/OpenVMS
QNX
VxWorks
Psion
Python has also been ported to the Java and .NET virtual machines
Getting Python
The most up-to-date and current source code, binaries, documentation, news, etc., is
available on the official website of Python https://www.python.org/
Installing Python
Downloading python
Click Python Download.
Installing
1. Double-click the icon labeling the file python-3.7.0.exe.
2. Click Run.
Setting up PATH
Programs and other executable files can be in many directories, so operating systems
provide a search path that lists the directories that the OS searches for executables.
In Mac OS, the installer handles the path details. To invoke the Python interpreter
from any particular directory, you must add the Python directory to your path.
1 PYTHONPATH
2 PYTHONSTARTUP
3 PYTHONCASEOK
4 PYTHONHOME
Running Python
There are three different ways to start Python −
Interactive Interpreter
You can start Python from Unix, DOS, or any other system that provides you a
command-line interpreter or shell window.
$python # Unix/Linux
or
python% # Unix/Linux
or
C:> python # Windows/DOS
1 -d
2 -O
3 -S
4 -v
5 -X
7 file
or
or
Macintosh − The Macintosh version of Python along with the IDLE IDE is
available from the main website, downloadable as either MacBinary or
BinHex'd files.
If you are not able to set up the environment properly, then you can take help from
your system admin. Make sure the Python environment is properly set up and
working perfectly fine.
CNN Algorithm
Notations:
W: input volume size
F: receptive field size of the Conv Layer
S: stride
P: amount of zero padding used
W, H, D: width, height and depth
Algorithm:
The Convolutional Layer:
The Conv layer is the core building block of a Convolutional Network that does most
of the computational heavy lifting. To summarize, the Conv Layer:
Accepts a volume of size W1×H1×D1
Pooling Layer:
Its function is to progressively reduce the spatial size of the representation to
reduce the number of parameters and computation in the network, and hence to
also control overfitting.
More generally, the pooling layer:
Accepts a volume of size W1×H1×D1
Estimating LOC for this project is difficult at estimation stages this project is
of innovative type project. Average estimation of this project is 10000 to
12000 line of code.
LOC based Estimation:
E = 3:2 (KLOC)1:05
E = 3:2 9:01:05to11:0 4:21:05
Directory 1.0-1.3
matching code
Testing 1.1-1.2
Total 9.0-10.11
3. RAM: 4GB
In appropriate dataset -To overcome this risk we are trying to use well
orga- nized and complete dataset.
When solving problems we have to decide the difficulty level of our problem.
There are three types of classes provided for that. These are as follows:
P class
NP-Hard Class
NP Complete Class
P Class:-
In computational complexity theory, P, also known as PTIME or
DTIME(nO(1)), is a fundamental complexity class. It contains all decision
problems that can be solved by a deterministic Turing machine using a
polynomial amount of computation time, or polynomial time.P class problems
are deterministic problems i.e. P class prob- lems can be solve by
deterministic Turing Machine.
Np Class:-
In computational complexity theory Equivalently, the formal definition of NP
is the set of decision problems solvable in polynomial time by a theoretical non-
deterministic Turing machine. This second definition is the basis for the
abbreviation NP, which stands for ”nondeterministic, polynomial time.”
However, the verifier-based defini- tion tends to be more intuitive and
practical in common applications compared to the formal machine definition.
The two definitions are equivalent because the algorithm for the machine
definition consists of two phases, the first of which consists of a
3. NP Hard Class:-
Informally, ”at least as hard as the hardest problems in NP” are called
as NP hard class problem.
6. Does the software engineering team have the right mix of skills? Ans:
Yes, we have.
The risks for the Project can be analyzed within the constraints of time and quality.
Impact
I Risk Description Prob
D abili Sc Q O
ty he u v
du al e
le it r
y a
ll
1 Deadline Risk me Lo H H
diu w ig i
m h g
h
2 Technical skill Risk me Lo H H
diu w ig i
m h g
h
3 Hardware Failure Risk hig hi m h
h gh e i
di g
u h
m
4 Accuracy Risk me me lo h
diu di w i
m u g
m h
Risk ID 2
Risk Technical skill risk
Descriptio
n
Category Requirements
Source Software Design Specification documentation review.
Probabilit Low
y
Impact High
Response Mitigate
Strategy Self study and Internet will be best source for
technology knowledge
Risk Identified
Status
Risk ID 3
Risk Hardware Failure
Descriptio
n
Category Requirements
Source Software Design Specification documentation review.
Probabilit Low
y
Impact High
Response Mitigate
Strategy Check whether Hardware is accurately fit or not
according to knowledge
Risk Identified
Status
Project tasks and their dependencies are noted in this diagrammatic form.
Figure 5.1: Task Network
The team structure for the project is identified. Roles are defined. Our
team have three members. We select this topic after discussing with each
other. All the members performing all the task whatever tasks are assign to
the members.
Name Role
The system will be available 100% of the time. Once there is a fatal
error, the system will provide understandable feed back to the user.
All data will be backed-up every day automatically and also the system
ad- ministrator can back-up the data as a function for him.
Reusability : Files of any type can be used by the system for any
number of times during transformation.
CHAPTER 7
SOFTWARE TESTING
Types of Testing:
Along with the type of testing also mention the approach to be followed for
the test- ing, that is, Manual Testing or Automated Testing. Use Automated
Testing Plan forplanning automation activities in details. The different types
of testing that may be carried out in the project are as follows:
Unit Testing:
Individual components are tested independently to ensure their quality.
The focus is to uncover errors in design and implementation, including
– Component interface
testing begins construction and testing with the lowest levels in the
program structure. Because modules are integrated from the bottom-
up,processing re- quired for modules subordinate to a given level is
always available and the need for test stubs is eliminated.
The focus is to uncover errors in:
System Testing :
The system software is tested as a whole. It verifies all elements mesh
properly to make sure that all system functions and performance are
achieved in the target environment. The focus areas are:
Validation Testing:
Validation can be defined in many ways, but a simple definition is that
suc- ceeds when software functions in a manner that can be reasonably
expected by the customer. Software validation is achieved through a
series of black- box tests that demonstrate conformity with
requirements. A test plan outlines the classes of tests to be conducted
and a test procedure defines specific test cases that will be used to
demonstrate conformity with requirements. Both the plan and procedure
are designed to ensure that all functional requirements are
GUI Testing:
Graphical User Interface (GUIs) present interesting challenges for
software engineers. Because of reusable components provided as part of
GUI devel- opment environments, the creation of the user interface has
become less time consuming and more precise. But, the same time, the
complexity of GUIs has
grown, leading to more difficulty in the design and execution of the test
cases. Because many modern GUIs have the same look and same feel, a
series of test cases can be derived.