Hand Sign Language Recognition using Deep
Learning
Under the guidance of
Ms. Rajeswari & Ms. Aishwarya
by
Sruthi Suresh(40290097)
Yashini. V (40290115)
June 2, 2024 School of Computing 1
Presentation Outline
• Introduction
• Motivation
• Literature survey
• Inferences from Literature Survey
• Objectives
• System Architecture
• Description of Software for Implementation
• Project Management Plan
• References
• Q&A
June 2, 2024 School of Computing 2
Introduction
• There have been several advancements in technology and a lot of
research has been done to help the people who are deaf and dumb.
Aiding the cause, Deep learning, and computer vision can be used
too to make an impact on this cause.
• This can be very helpful for the deaf and dumb people in
communicating with others as knowing sign language is not
something that is common to all, moreover, this can be extended to
creating automatic editors, where the person can easily write by just
their hand gestures.
June 2, 2024 School of Computing 3
Objectives
•Sign Language is a language that uses hand gestures, facial
expressions, and body movements for communication. Sign
language consists of either word-level signs or fingerspelling. It is
the only communication means for the deaf-dumb community. But
hearing people never try to learn sign language. So deaf people
cannot interact with normal people without a sign language
interpreter. This causes the isolation of deaf people in society. So, a
system that automatically recognizes sign language is necessary.
• The implementation of such a system provides a platform for the
interaction of hearing-disabled people with the rest of the world
without an interpreter. In this paper, we propose a method for the
automatic recognition of fingerspelling in American sign language.
The proposed method uses digital image processing techniques
and an artificial neural network for recognizing different signs.
June 2, 2024 School of Computing 4
Motivation
• The project idea emerged from first hand experience of surfing huge
database of videos to get the information required.
• The motivation was to decrease the time spent on searching for
information in the web and making it easier to learn.
• I was already aware of an existing summarizing tool I used- “liner”
• Liner is a chrome extension that summarized online articles and websites.
It is highly proficient and extracts the required information.
• As a data analyst I was curious to learn the concepts behind such
algorithms and try and implement the same to videos.
• This idea motivated me to learn such algorithms and pose to try and
implement and optimize existing solutions.
June 2, 2024 School of Computing 5
Literature Review
YEAR OF
AUTHOR DESCRIPTIONS PROS CONS
PUBLICATION
Vardhese and 2015 Summarized video by using Reduces 55% The overall
Nair SSI and Stroboscopic effect
of the original accuracy is
video. questionable.
Dataset used: Random videos
Miniakhmetova 2015 User preference analysis for Video Dataset used was
and Zymbler summarizing personalized summaries are biased and
visual data.
generated with performed will
Dataset used: Personalized user preferences only on it.
videos estimations
Srinivas 2016 A rank- based approach is Frame blocks Not cost
used to score, rank and features method effective and
eliminate the frames.
improved by demands high
1.8%
Dataset used: Open video project computation.
June 2, 2024 School of Computing 6
Literature Review
YEAR OF
AUTHOR DESCRIPTIONS PROS CONS
PUBLICATION
Almedia 2010 Video summarization by Relatively new Model is hardware
macroblocks and color method that dependent that
histogram
brought new having high
computational
Dataset: Open Video Project perspective power.
Wang and Feng 2019 Two level hirachial hidden Ignores events The model
Markov Model (HHHM) that aren’t showed low
précising rushes video based
events, objects and motion
motion accuracy
insensitive.
Dataset: TRECVID BCC video
datasets
Chasanis 2008 Video summarization by shot Spectral Over
boundary detection and clustering is segmentation
detection and spectral
clustering.
dynamic and occurs in
poses as a good continuously
Dataset: TV series/ Sports technique changing scenes.
June 2, 2024 School of Computing 7
Inferences from Literature Survey
• The literature survey portrays the new methodologies and techniques
used to summarize
• [Link]
cases/
• [Link]
learning-algorithm
June 2, 2024 School of Computing 8
System Architecture
1. Creating the dataset for sign 2. A live feed from the 3. When the hand is present in the
ROI, we start to save the image of the
language detection webcam is captured and ROI in the train and test set
detects every frame respectively for the letter or number
we are detecting it for.
Web Browser Computation API
5. Display the 4. Perform Transcript
summarized text on the summarization and return it
extension as a HTTP Response
June 2, 2024 School of Computing 9
Description of Software for Implementation
STEP 1
Make a back-end application directory with the files Requirements:
[Link] and [Link]. Flask, youtube_transcript_api, transformers[torch]
STEP 2
[Link] the YouTube API to receive the video transcript
Creating a function that accepts YouTube video id
[Link] the data from the response
returns parsed transcript as output
June 2, 2024 School of Computing 10
Description of Software for Implementation
Perform Text Summarization:
STEP 3 Should receive YouTube transcript as an input parameter
return summarized as output.
1. Extractive Summarization
Choice: 6 accepted values
2. Abstractive Summarization
Gensim, Spacy, NLTK, Sumy-LSA, Sumy-Luhn, Sumy-text-rank
STEP 4
Generate the summarized transcript by executing by executing
Create /Summarize/ API Endpoint Create a Flask API Route
transcript generation function followed by executing transcript
with GET HTTP request with URL
summarizer function.
June 2, 2024 School of Computing 11
Description of Software for Implementation
Getting started with Web Extension:
STEPCreate
5 a web application directory containing input
field to get the youtube URL, a drop down for
They are built on web technologies such as HTML, CSS selecting summary algorithm and a summarize
and Javascript button.
STEP 6
Display Summarized Transcript This endpoint displays the transcripted summary from
the backend Summarizer API call and the text is
displayed once received.
June 2, 2024 School of Computing 12
References
1] J. Varghese, K. R. Nair, ―An Algorithmic Approach for General Video Summarization‖. In
2015 Fifth International Conference on Advances in Computing and Communications (ICACC),
pp. 7-11, 2015.
2] K. Muhammad, T. Hussain, and S.W. Baik, ―Efficient CNN based summarization of
surveillance videos for resource-constrained devices‖. Pattern Recognition Letters, 130, pp.
370-375, 2020.
3] M. Ma, S. Mei, S. Wan, Z. Wang, D. D. Feng. ―Robust video summarization using
collaborative representation of adjacent frames‖. Multimedia Tools and Applications, 78(20),
pp.28985-29005, 2019.
4] P. K. Lai, M. Décombas, K. Moutet, R. Laganière, ―Video summarization of surveillance
cameras‖. In 2016 13th IEEE International Conference on Advanced Video and Signal Based
Surveillance (AVSS), pp. 286-294, 2016.
5] S. Mei, M. Ma, S. Wan, J. Hou, Z. Wang, D.D. Feng, ―Patch based Video Summarization
with Block Sparse Representation‖. IEEE Transactions on Multimedia, 2020
June 2, 2024 School of Computing 13
References
6] S. Khan, S. Pawar. ―Video summarization: survey on event detection and summarization
in soccer videos‖. International Journal of Advanced Computer Science and Applications, 6(11),
2015.
7] A. Workie, R. Sharma, Y. K. Chung. ―Digital Video Summarization Techniques: A Survey‖, 2020.
8] G. Evangelopoulos, K. Rapantzikos, A. Potamianos, P. Maragos, A. Zlatintsi, Y. Avrithis, ―Movie
Summarization Based on Audio-Visual Valiency Detection‖. In: IEEE Intl Conf. Image Processing
(ICIP), San Diego, CA,2008
9] S. Patil, D. Phalke, International Journal for Research in Applied Science & Engineering
Technology (IJRASET) Retrieved from [Link] 2019.
10] Almeida, R. D. S. Torres, N. J. Leite, ―Rapid video summarization on compressed video‖. In
2010 IEEE International Symposium on Multimedia , pp. 113-120,2010
June 2, 2024 School of Computing 14
