A Transformer based Dynamic Hand Gesture

Recognition
Ankit Kumar
Supervisor : Dr. Sunil Kumar
ABV-IIITM, Gwalior
August 1, 2022

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022

 Table of Contents

❏ Introduction

❏ Objective

❏ Literature Review

❏ Research Gap

❏ References

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022

 Introduction

With the recent advancements in the field of Machine Learning & Artificial Intelligence, there
has been an upcoming trend of tools and robots to perform activities previously performed by
humans.For this, the machine needs to understand the requirements and language of the user
and what the user wants to communicate.

This was done in the past by giving a fixed set of instructions to the computer, but recent
emphasis has been on making these machines smarter or “intelligent” to perform even more
complex activities by adapting and learning as per requirement.

This has led to a burst of research recently in the field of Human Computer Interaction.

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022

 Communication with machines using speech and written text has been widely explored in the
past with already existing virtual assistants such as Siri and Alexa already a major part of our
lifestyle now.

However, the work on the use of actions or gestures to communicate with a machine to signal it
to perform a set of activities is booming. The following work aims at solving this problem and
classifying hand gestures or actions to perform certain tasks.

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022

 Objective

❏ To develop a method for continuous hand gesture recognition.

❏ To explore different state-of-art methods for hand gesture recognition.

❏ To come up with architecture for dynamic hand gesture recognition.

❏ Address the temporal and sequential nature of dynamic gestures.

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022

 Literature Review
Learning spatio-temporal features is critical for performance to be stable in human hand
gesture. Many methods have been proposed for hand gesture recognition in recent
years.Munasinghe proposed a feed forward neural network based approach for identifying four
gestures [2]. In this approach, each frame is preprocessed using multiple image processing steps
like background subtracting along with applying blur. Each frame has deviation which
concatenate to form a single Motion History Image(MHI). When the deviation reaches a
threshold value then MHI is sent to neural network for classification which then return
probabilities for each type of gesture category and if the maximum probability exceeds 0.8, it
is considered as a correctly classified gesture. Bao et al. [3] proposed a two-dimensional
CNN model for recognition of gestures. A nine-layer CNN is used to directly categorize hand
gesture present in the images without preprocessing segmentation of the region of interest.

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022

 The presented technique is able classify seven different types of hand gestures in real-time.

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022
 In [1], the architecture consists of a 3D-CNN followed by an LSTM network which learns both
spatial and temporal features of all video frames under complex background and lighting
conditions. The proposed model was trained on a subset of 20BN-jester dataset that contained
15 classes with unique hand gestures.

In [4], the author proposed a recurrent 3D convolutional neural network (R3DCNN) for dynamic
hand gesture recognition.The architecture consists of a deep 3D-CNN for spatio-temporal
feature extraction, a recurrent layer for global temporal modeling, and a softmax layer for
predicting class-conditional gesture probabilities.

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022
 Research Gap

In the state-of-art approach, it takes a n sequential frames containing the whole gesture. The
detection of beginning and the end gesture is not taken in account and the distinction between
gesture and no gesture sequences are also not taken into account.Basically it uses the sliding
window technique to get the gesture.

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022

 References

1. Rehman, Muneeb & Ahmed, Fawad & Khan, Muhammad & Tariq, Usman & Alfouzan, Faisal
& Alzahrani, Nouf & Ahmad, Jawad. (2021). Dynamic Hand Gesture Recognition Using 3D-
CNN and LSTM Networks. Computers, Materials and Continua.
10.32604/cmc.2021.019586.
2. Munasinghe, Nuwan. (2018). Dynamic Hand Gesture Recognition Using Computer Vision
and Neural Networks. 10.1109/I2CT.2018.8529335.
3. Bao, Peijun & Maqueda, Ana & Del-Blanco, Carlos & García, Narciso. (2017). Tiny hand
gesture recognition without localization via a deep convolutional network. IEEE
Transactions on Consumer Electronics. 63. 251-257. 10.1109/TCE.2017.014971.

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022

 4. Molchanov, P., Yang, X., Gupta, S., Kim, K., Tyree, S. and Kautz, J., 2016. Online detection
and classification of dynamic hand gestures with recurrent 3d convolutional neural
network. In Proceedings of the IEEE conference on computer vision and pattern
recognition (pp. 4207-4215).

Ankit Kumar(2019IMG-007) Dynamic Hand Gesture August 1, 2022