A Project Report on

Sign language Recognition using

Machine Learning

Submitted by

Arati M. Prajapati SS20CO047

Yukta R. More SS20CO065

Sanika S. Khade SS20CO038

Amisha B. Sawant SS20CO005
Megha M. Pawar SS20CO006

In partial fulfillment of the requirements for

Diploma in Computer Engineering
2022-2023

Under the guidance of

Prof. P.S. Sadafule

Department of Computer Engineering

Government Polytechnic, Mumbai Mumbai-
400051
 Department of Computer Engineering
Government Polytechnic, Mumbai
Mumbai-400051

CERTIFICATE

This is to certify that following students have successfully and satisfactorily completed
the project on Sign Language Reorganization using Machine Learning presented their
report in the partial fulfillment of requirement for Diploma in Computer Engineering from
Government Polytechnic, Mumbai under the guidance of Prof. P.S. Sadafule in the academic
year 2020-2022.

Arati M. Prajapati SS20CO047

Yukta R. More SS20CO065
Sanika S. Khade SS20CO038
Amisha B. Sawant SS20CO005
Megha M. Pawar SS20CO006

Project Guide External Examiner

 Head Of the Department
Principal

ACKNOWLEDGEMENT

It is always a difficult job to acknowledge all those who have been of tremendous help
in the development of any work. In spite of that, we honestly thank all those who had put in
tremendous efforts in guiding and helping us to successfully build this project named “Sign
Language Reorganization using ML”. We take this opportunity to thank each and every
one for their co-operation and support.

First and firmly, we are grateful to (principal, Government Polytechnic Mumbai) for
allowing us to do this project collectively and with advanced techniques.

We express our sincere gratitude towards Prof. Neha. H. Vachani (Head of Computer
Department).

We heartly thank Prof. P. S. Sadafule our project guide for their valuable guidance
throughout, without which the project would not have stood its ground as it has now. We
would even like to thank people who have directly or indirectly helped us with the making of
the project and their support.

We are thankful to and fortunate enough to get constant encouragement, support, and
guidance from all Teaching staffs of Computer department, GPM which helped us in
successfully progressing with our project work.

ARATI M. PRAJAPATI -SS20CO047

YUKTA R. MORE-SS20CO065
 SANIKA S. KHADE -SS20CO038

AMISHA B. SAWANT-SS20CO005

MEGHA M. PAWAR-SS20CO006

Index
SR.NO CONTENTS PAGE
. NO.

1 INTRODUCTION
.
1.1 Objectives
1.2 Problem Statement
1.3 Literature survey

2 Feasibility Study
.
2.1 project Scope

2.2 Approach

3 Project Planning
.
3.1 Scope Schedule and Cost
And Risks
3.2 Communication, Resources

3.3 Change in Plan

4 Requirement Analysis
.
 4.1 Hardware Requirements
4.2 Software Requirements

5 System Design
.
5.1 System Architecture

6 Coding
. Collection.py
6.1 data y

6.2 test.p

6.3 api

7 Implementation
.

8 Testin
. g

9 Output/Screenshots
.

1 Bibliography
0

ABSTRACT
Sign Language is mainly used by deaf (hard hearing) and dumb people to
exchange information between their own community and with other people. It is a
language where people use their hand gestures to communicate as they can’t speak or
hear. Sign Language Recognition (SLR) deals with recognizing the hand gestures and
continues till text or speech is generated for corresponding hand gestures. Here hand
gestures for sign language can be classified as static and dynamic. However, static
hand gesture recognition is simpler than dynamic hand gesture recognition, but both
recognition is important to the human community. We can use Deep Learning
Computer Vision to recognize the hand gestures.

Where the model will learn to recognize the hand gestures images. Once the model
Successfully recognizes the gesture the corresponding English text is generated and
then text can be converted to speech. This model will be more efficient and hence
communicate for the

deaf (hard hearing) and dump people will be easier. In this, we will discuss how Sign
Language Recognition is done using Machine Learning.
 Chapter – 1
INTRODUCTION

INTRODUCTION
Deaf (hard hearing) and dumb people use Sign Language (SL) as their primary
means To express their ideas and thoughts with their own community and with other
people with hand and body gestures. It has its own vocabulary, meaning, and syntax
which is different from the spoken language or written language. Spoken language is
a language produced by articulate sounds mapped against specific words and
grammatical combinations to convey meaningful messages. Sign language uses visual
hand and body gestures to convey meaningful messages. There are somewhere
between 138 and 300 different types of Sign Language used around globally today. In
India, there are only about 250 certified sign language interpreters for a deaf
population of around 7 million. This would be a problem to teach sign language to the
deaf and dumb people as there is a limited number of sign language interpreters exits
today. Sign Language Recognition is an attempt to recognize these hand gestures and
convert them to the corresponding text or speech. Today Computer Vision and Deep
Learning have gained a lot of popularity. Using Deep Learning algorithms and Image
Processing we can be able to classify these hand gestures and able to produce
corresponding text.

1.1 Objectives
The main objectives of this project are to contribute to the field of automatic sign
language recognition and translation to text or speech. In our project, we focus on static
sign language hand gestures. This work focused on recognizing the hand gestures which
includes 26

English alphabets (A-Z) and 10 digits (0-9). By using ‘PyCharm’ and using ML with the help
oh python libraires we executed the code to show the hand gesture.

1.2 PROBLEM STATEMENT

Speech impaired people use hand signs and gestures to communicate. Normal
people face difficulty in understanding their language. Hence there is a need of a system
which recognizes the different signs, gestures and conveys the information to the normal
people. It bridges the gap between physically challenged people and normal people.
1.3 LITERATURE SURVEY

This works focuses on static fingerspelling in American Sign Language A

method for implementing a sign language to text/voice conversion system without
using handheld gloves and sensors, by capturing the gesture continuously and
converting them to voice. In this method images were captured for recognition. The
design of a communication aid for the physically challenged.

Android Studio: Android Studio Dolphin | 2021.3.1

Android Studio is the official IDE for Android development, and includes everything you
need to build Android apps.

This page lists new features and improvements in the latest version in the stable channel,

Android Studio Dolphin. You can download it here or update to it inside Android

Studio by clicking Help > Check for updates (Android Studio > Check for updates on
macOS).

Features:

Code and iterate faster than ever

Based on IntelliJ IDEA, Android Studio provides fast turnaround on your coding and running
workflow.

Code with confidence

At every step, Android Studio helps ensure that you're creating the best code possible.

Configure builds without limits

Android Studio's project structure and Gradle-based builds provide the

flexibility you need to generate APKs for all device types.

Create rich and connected apps

Android Studio knows not all code is written in Java, and not all code runs on the user's
device.

Eliminate tiresome tasks

Android Studio provides GUI tools that simplify the less interesting parts of app
development.

Tensorlight: Tensor Light is a high-level framework for TensorFlow-based machine

intelligence applications. It reduces boilerplate code and enables advanced features that
are not yet provided out-of-the-box.

Key Features
To highlight the advanced features of Tensor Light, an incomplete list of some main
functionalities is provided that are not shipped with TensorFlow by default, or might
even be missing in other high-level APIs. These include:
• Transparent lifecycle management of the session and graph definition.
• Abstraction of models and datasets to provide a reusable plug-and-play support.
 • Effortless support to train a model symmetrically on multiple GPUs, as well as prevent
TensorFlow to allocate memory on other GPU devices of the cluster.
• Train or evaluate a model with a single line of code.
• Abstracted, runtime-exchangeable input pipelines which either use the simple feeding
mechanism with NumPy arrays, or even multi-threaded input queues.
• Automatic saving and loading of hyperparameters as JSON to simplify the evaluation
management of numerous trainings.
• Ready-to-use loss functions and metrics, even with latest advances for perceptual
motivated image similarity assessment.
• Extended recurrent functions to enable scheduled sampling, as well as an
implementation of a ConvLSTM cell.
• Automatic creation of periodic checkpoints and Tensor Board summaries.
• Ability to work with other higher-level libraries hand in hand,

TensorFlow:
TensorFlow is a free and open-source software library for dataflow and
differentiable programming across a range of tasks. It is a symbolic math l library and is
also used for machine learning applications such as neural networks. It is used for both
research and production at Google.

Features:

• TensorFlow provides stable Python (for version 3.7 across all platforms) and C APIs;
and without API backwards compatibility guarantee: C++, Go, Java, JavaScript and
Swift (early release).
• Third-party packages are available for C#,. "New language support should be built on
top of the C API. However, not all functionality is available in C yet." Some more
functionality is provided by the Python API.
• Application among the applications for which TensorFlow is the foundation, are
automated image-captioning software, such as Deep Dream.
OpenCV:
OpenCV (Open-Source Computer Vision Library) is a library of programming
functions mainly aimed at real-time computer vision.[1] Originally developed by Intel,
it was later supported by Willow Garage then Itrez (which was later acquired by Intel
[2]). The library is cross-platform and free for use under the open-source BSD license.

OpenCV's application areas include:

• 2D and 3D feature toolkits
• Facial recognition system
• Gesture recognition
• Human–computer interaction (HCI)
• Mobile robotics
• Motion understanding
• Object identification

List of free and open-source software packages

• OpenCV Functionality
• Image/video I/O, processing, display (core, improv, high-up)
• Object/feature detection (obj detect, features2d, nonfee)
• Computational photography (photo, video, supers)
• Machine learning & clustering (ml, Flann)
• CUDA acceleration (gnu)

Image-Processing:

Image processing is a method to perform some operations on an image, in order to get

an enhanced image and or to extract some useful information from it. If we talk about
the basic definition of image processing, then “Image processing is the analysis and
manipulation of a digitized image, especially in order to improve its quality”.

keras:
Keras is an open-source neural-network library written in Python. It can run on top of
TensorFlow, Microsoft Cognitive Toolkit, R, Theano, or Plaid ml. Designed to enable
fast experimentation with deep neural networks, it focuses on being user-friendly,
modular, and extensible.

Features:
1)Large Dataset:

Keras contains a large pre-defined dataset. It provides you a variety of datasets. You can use
this dataset to be directly importing and loading it.
2) Pre-processing of Data:
Keras provides you several functions for the pre-processing of data.
3)Modularity:
Keras is modular, the model you train and use this model later by loading it.

NumPy:
It is a library for the Python programming language, adding support for large,
multidimensional arrays and matrices, along with a large collection of high-level
mathematical functions to operate on these arrays.

Features:

• NumPy targets the Python reference implementation of Python, which is a non-

optimizing bytecode interpreter. Mathematical algorithms written for this version of

Python often run much slower than compiled equivalents.

 CHAPTER 2
Feasibility Study

Feasibility Study

A feasibility study is an analysis that takes all a project’s relevant factors into account—
including economic, technical, legal, and scheduling considerations—to ascertain the
likelihood of completing the project successfully. Project managers use feasibility studies to
discern the pros an cons of undertaking a project before they invest a lot of time into it.

2.1 Project Scope

This project base on the Machine Learning and on android studio to help the people who are
deaf and dump. By approaching this application in future or in current life people can easily
communicate with other peoples.

By doing this project we opened the door for the people who can’t communicate properly or
adjust with world. Using this application people will enjoy and learn something new
language.

2.2 Approach
This application will be using for native approach of development. The application is created
using Machine learning &Python , recognition is achieved by TensorFlow, OpenCV, Cvzone
for desktop and for mobile app we use android studio. Recognize the user which show the
sign and display on screen. Mostly usable for all people who can interested to learn sign
language, detect sign language and help deaf and dump people communicating with each
other.

CHAPTER 3
 Project Planning

project planning

Project planning is a discipline addressing how to complete a project in a certain time period,
usually with defined stages and designated resources. One view of project planning divides
the activity into these steps:

⚫ setting measurable objectives

⚫ identifying deliverable
⚫ scheduling
⚫ planning tasks

Scheduling of the project given below:

Sr. Date Day Present Subjects
No Members

1. 20/07/2022 Wednesday All Introduction to Third Year Project

2. 28/07/2022 Thursday All Group Selected

3. 29/07/2022 Friday All Abstract Submitted

4.. 05/08/2022 Friday All Guide Allocated

5. 12/08/2022 Friday 4 members Abstract Rejected

6. 26/08/2022 Friday All Project Topic Selected

7. 17/09/2022 Saturday All Project Detail Research

8. 29/09/2022 Thursday 3 members Synopsis Submitted

9. 01/10/2022 All members Coding Started

10. 05/10/2022 All members Working on Capture Images Frame

11. 06/10/2022 All members Data Collection started

12. 10/10/2022 All members Data Collection Halfway

13. 13/10/2022 All members Data Collection Completed

14. 15/10/2022 All members Show output until now completed
project (45% project completed)

15. 21/10/2022 All members Addition in data

16. 28/10/2022 New features added

17

18.
 CHAPTER 4
Requirement Analysis

Requirement Analysis

We present a sign language recognition that can be helpful for the deaf and dumb
people while communicating with others. In the sign language recognition, we
create sign detector which detects numbers from 0 to 9 and hand gesture
including the alphabets and words.

Requirement Specification:
➢ Hardware Requirements ➢
Software Requirements

4.1 Hardware Requirements:

The hardware used for the development of the project:

• Processor: AMD Ryzen 5 4600H with Radeon Graphics 3.00GHz

• RAM: 8.00 GB
• Devices: Laptop, Laptop Webcam
4.2 Software Requirements:
The software used for the development of the project:
• IDE such as PyCharm (Community Version 2022.2.3) will be used for coding.
• Android Studio.

Libraries/Packages: Version Description

OpenCV python 4.5.4.60 • It is used for image processing and

performing computer vision task.

TensorFlow, 2.9.1 • Used to create deep learning model.

Tensorlight
Cvzone 1.5.6 • It makes easy to run image processing and
AI function.

Mediapipe 0.8.10.1 • media pipe is a pipeline framework use

to build custom machine learning
solutions for live and streaming media
CHAPTER 5
System Design
 System designing
System designing is the process of design/draw the architecture, modules and components,
the data goes through that system. Though the design process it becomes easier to
understands the software/ products flow to working.

Let’s draw the System Architecture of our Sign Language Recognition to understand
clearly:
 Fig.1: Block Diagram of System

As shown in Figure 1, the project will be the entire system is divided into 3 parts: -

1) Data Processing

2) Training

3) Classify Gesture

Let’s discuss in detail: -

1)Data Processing:
The DataCollection.py script contains functions to load the images and save it on data as
NumPy arrays into file storage. in short Data Collection script is use for webcam through
which we can capture the images and save it on “Data” Folder. After that it will go for
training.

2)Training:

The training loop for the model is contained in train model. The model is trained with n
number of saved image and that images will be save on Models folder in which two file will
be generated “keras_models.h5” and “labels.txt” in which all the input value predefined.

3) Classify Gesture:

As we have used supervised machine learning algorithm in our programs. After training, the
users give the input, and the system will process it or recognize it and gives the output on the
screen.

let’s take it more simple-

through this diagram you can clearly understand how the system is worked flow.
 CHAPTER 6
Coding CHAPTER
7

Implementation

Android Stuido:

Android architecture contains different number of components to support any android device
needs.Android software contains an open-source Linux Kernel having collection of number
of C/C++ libraries which are exposed through an application framework services.Among all
the components Linux Kernel provides main functionality of operating system functions to
smartphones and Dalvik Virtual Machine (DVM) provide platform for running an android
application.

Architecture:

The main components of android architecture are following:-

Applications

Application Framework

Android Runtime

Platform Libraries

Linux Kernel
TensorLight Architecture

From an architectural perspective, the framework can be split into three main components.

First, a collection of utility function that are unrelated to machine learning.

Examples are functions to download and extract datasets, to process images and videos, or
to generate animated GIFs and videos from a data array, to name just a few.

Second, the high-level library which builds on top of TensorFlow. It includes several
modules that either provide a simple access to functionally that it repeatedly required when
developing deep learning applications, or features that are not included in TensorFlow yet.

For instance, it handles the creation of weight and bias variables internally, offers a bunch of
ready-to-use loss and initialization functions, or comes with some advanced visualization
features to display feature maps or output images directly in an IPython Notebook.
Third, an abstraction layer to simplify the overall lifecycle, to generalize the definition of a
model graphs, as well as to enable a reusable and consistent access to datasets.

Working of TensorFlow
TensorFlow allows you to create dataflow graphs that describe how data moves through a
graph. The graph consists of nodes that represent a mathematical operation. A connection or
edge between nodes is a multidimensional data array. It takes inputs as a multi-dimensional
array where you can construct a flowchart of operations that can be performed on these
inputs.

Architecture of TensorFlow:
 ➢ Input Layer

The input layer accepts large volumes of data as input to build the network. The data can be
in the form of text, image, audio, etc.
 ➢ Hidden Layer:

This layer processes data by performing complex computations and carries out feature
extraction. As part of the training, these layers have weights and biases that are
continuously updated until the training process is complete.

➢ Output Layer

The output layer generates predicted output by applying suitable activation functions. The
output can be in the form of numeric or categorical values.

For example, if it is an image classification application, it tells us which class a particular

image may belong to. The input can be multiple images, such as cats and dogs. The output
 Flowchart of Sign Language Recognition

can be in the form of binary classification like the number zero for the dog and the number
one for the cat
Explanation: -

Step1: - In this phase we run the program and it will open the “Webcam”.

Step2: -It invokes the OpenCV, Keras, and TensorFlow such library for processing the
images.

Step3: - This step consists of digitize the information from the user to make it usable to
classification algorithm.

Step4: - It will re-process on the datasets.

Step5: - It is the process of transforming raw data into numerical features that can be
processed while preserving the information in the original datasets.

Step6: - It compare two different sets of data and matching them with input data. If data is
matched, it will display the outputs otherwise it will repeat the process until the data cannot
be matched.
 CHAPTER 8
Testing

TESTING

Testing is the process of checking that software product or application are

matches to the expected requirement or not and identifying the correctness of
software by considering its all attributes (Reliability, Scalability, Portability,
Reusability, Usability).

The purpose of testing is to discover the error and every conceivable fault or
weakness in a work product.

Software testing is an important element of the software quality assurance

and evaluate work products such as user requirements, design, and code.

Testing Objectives:
Testing objectives can serve as a several rules they are:

• To find failures and defects.

• To provide sufficient information to the stakeholder to make decisions regarding
the level of quality of the test objects.

Types of testing:
➢ Unit Testing:
Unit testing is done on individual models as they completed and becomes
executable. It helps to detect flaw in code which is more difficult to find in
later in testing process. Unit testing is different from and should be
preceded by other techniques including:
• Inform debugging
• Code Inspection

➢ Black Box Testing:

This type of testing approach focusses on the input that goes into the software and
that produced output.
This testing is used to find the error of following categories:
• Interface and performance errors
• Initialization and termination of errors
• The logical flow of data is not checked
• In this testing the only Output is checked for correctness

➢ White Box Testing:

This testing approach allow tester to verify the inner working of software
systems (code, infrastructure, integration with external system).
It has been used to generate test cases in the following cases:
• Execute internal data structure to ensure their validity
• Guarantee that are all independent paths have been executed

➢ Integration Testing:

Integration testing is the second level of the software testing process comes
after unit testing.
 In this testing, units or individual components of the software are tested in a
group. The software is developed with a few software modules that are
coded by different coders or programmers.

➢ System Testing:

It is a level of testing that validates the complete and fully integrated software product.
The purpose of a system test is to evaluate the end-to-end system specifications.
Usually, the software is only one element of a larger computer -based system.
Ultimately, the software is interfaced with other software/hardware system.

Test Cases:
Test Test Case Input Data Expected Actual Result Remar
Case Objectives /Description Result k/
Status
ID

TC 1 Loading Initializing the Loaded the The Datasets was

Datasets trained Datasets Datasets without loaded
and load it into successfully Pass
error

TC 2 Convert Capture the Image words The Conversion

image to image and and alphabets of of image to
Pass
alphabets convert it into the captured hand alphabets and
and words words and gesture words
alphabets successfully
TC 3 Recognize Image Words and Label Recognition of
the hand Alphabets that the hand gesture
gesture successfully Pass
contain hand
objects

CHAPTER 9
Output/Screenshots
 CHAPTER 10
References /Bibliography

References /Bibliography

1.Thakur, Pujan Budhathoki, Sarmila Upreti, Shirish Shrestha, Subarna Shakya, “Real Time Sign Language
Recognition and Speech Generation,” Journal of Innovative Image Processing(Vol.02/No.02) Published on
01-02-2020.

Aman Pathak, Avinash Kumar, Priyam, Priyanshu Gupta and Gunjan Chugh, “Real Time Sign Language
Detection,” International Journal for Modern Trends in Science and Technology 2022, 8 pp. 32-37.

Reddygari Sandhya Rani , R Rumana , R. Prema,” Sign Language Recognition for The Deaf and Dumb,”
International Journal of Engineering Research & Technology (IJERT), Volume 10, Issue 10 (October 2021).

S. M. Prajapati, V. V. Tambe and D. D. Bendre, "SMART GLOVE THAT TRANSLATE SIGN

LANGUAGE INTO TEXT AND SPEECH," Accepted and presented by Electronics Engineering, Feb 2022.
[hardwere]

Anbarasi Rajamohan, Hemavathy R., Dhanalakshmi M.,” Deaf-Mute Communication Interpreter,”

International Journal of Scientific Engineering and Technology Volume 2 Issue 5, pp : 336-341 1 May
2013.[hardware]

Other references:
https://www.eecg.utoronto.ca/~jayar/mie324/asl.pdf

https://ieeexplore.ieee.org/document/7916786

https://babysignlanguage.com/dictionary/ https://data-flair.training/blogs/sign-

language-recognition-python-ml-opencv/

http://cs231n.stanford.edu/reports/2016/pdfs/214Report.pdf