Gesture Control Gaming
Tarinee Prasad Sahoo Kumar Mapanip Saheb Sahoo
Department of Computer Science and Engineering
National Institute of Technology Rourkela
Rourkela, India
116CS0224@nitrkl.ac.in
Abstract—This is a project implementing concepts of deep
learning and image processing to design a gesture based system
which uses various hand gestures to mimic keyboard commands.
These virtual commands will be used as the input for a game.
The game will be designed using PyGame module of python.
I. I NTRODUCTION
The main aim behind this project is to make an interaction
between human and computer using various applications run-
ning on computer by aiming basic shapes made by hand.Our
hand movements have an important role while interacting with
other people, as they convey very rich information in many
ways. Gesture recognition could be a quite sensory activity
computing program that permits computers to capture and
interpret human gestures as commands. Gesture Recognition
is largely the power of a computer to grasp gestures and
execute commands supported those gestures . Many companies
are working on gesture recognition system . Most of them
are working with additional sensors like wireless gloves with
sensors or the most popular Microsoft Kinect which uses
a Depth camera which basically consists of some system
that segment the body , and based on the segmentation they
produce the getsure . On the contorary we decided to feed a
neural network with a large amount of data and seeing whether
we can achieve the result without using much of hardware .
We only need a laptop camera to build a gesture recognition
system .
A. Why gesture?
Gesture is a natural form of expression and has been a vital
mode of communication for human.Building a gesture based
recognition system would be the first step for a computer
to understand human body language. Thus building a bridge
between human and computer interaction.
B. How Gesture Recognition works?
Gesture recognition is an alternate technique for provid-
ing period of time knowledge to a computer rather than
typewriting with keys or sound on barely screen, a motion
sensing element perceives and interprets movements because
the primary supply of knowledge input.The recognized gesture
can be used as data input for games. The camera feeds
image knowledge into a sensing device(camera) connected
Department of Computer Science and Engineering
National Institute of Technology Rourkela
Rourkela, India
716CS1046@nitrkl.ac.in
to the computer. The Programmed package then identifies
meaningful gestures from a planned gesture library wherever
every gesture is mapped to a associated command.The package
then associates every realtime gesture, interprets it , uses the
library to spot meaningful gestures that match with the given
gesture. Once the gesture has been understood, classification
is performed.
C. Stages of Processing
There are three main stages in working of a search engine,
namely
Data Collection
Data Preprocessing
Deep Learning
Score Computation
D. Related Work
Gesture Recognition is essentially the power of a pc to grasp
gestures and execute commands supported those gestures.
Most shoppers ar familiar with the conception through Wii
match, X-box and PlayStation games like Just Dance and
Kinect Sports.
II. A LGORITHMS U SED AND W ORKING
A. Adaptive Thresholding
Adaptive thresholding generally takes a grayscale or color
image as input and, within the simplest implementation, out-
puts a binary image representing the segmentation. for every
picture element within the image, a threshold needs to be
calculated. If the picture element price is below the brink
it’s set to the background price, otherwise it assumes the
foreground price.
There are two prime approaches to finding the threshold:
(i) the Chow and Kaneko approach and (ii) local thresholding.
the idea behind each strategies is that smaller image regions
are a lot of probably to possess around uniform illumination,
therefore being a lot of appropriate for thresholding. Chow
associate degreed Kaneko divide a picture into an array of
overlapping subimages and so notice the optimum threshold
for every subimage by investigation its bar chart. the brink
for every single picture element is found by interpolating the
results of the subimages. the disadvantage of this technique
is that it’s machine pricey and, therefore, isn’t acceptable for
period applications.
 An alternative approach to finding the native threshold
is to statistically examine the intensity values of the native
neighborhood of every picture element. The datum that is most
acceptable depends mostly on the input image. easy and quick
functions embrace the mean of the native intensity distribution,
T = Mean
the median value,
T = Median
or the mean of the minimum and maximum values,
T = (Max+Min)/2
Fig 1: Adaptive Thresholding
B. Convolutional Neural Network
Convolutional Neural Network is a category of deep neural
networks, most ordinarily applied to analyzing visual imaging.
CNNs use a variation of multilayer perceptrons designed to
want borderline preprocessing. they’re conjointly called shift
invariant or space invariant artificial neural networks (SIANN),
supported their shared-weights architecture and translation
unchangingness characteristics. Convolutional networks were
inspired by biological processes in this the property pattern be-
tween neurons resembles the organization of the animal visual
area. Individual plant tissue neurons respond to stimuli solely
in an exceedingly restricted region of the visual view called
the receptive field. The receptive fields of various neurons
partly overlap specified they cowl the entire visual view. CNNs
use comparatively very little pre-processing compared to al-
ternative image classification algorithms. this implies that the
network learns the filters that in ancient algorithms were hand-
engineered. This independence from previous information and
human effort in feature style could be a major advantage.
Fig 2: CNN
C. CNN Parameters
First layer : Convolution layer with 5 * 5 stride with relu
activation function
Second layer : Maxpooling layer with 2 * 2 pooling window
Third layer : Convolution layer with 5 * 5 sliding with relu
activation function
Fourth layer : Maxpooling layer with 2 * 2 pooling window
Flattening of outputs from fourth layer
Fifth layer : Full connection layer with 256 input nodes and
relu Activation
Sixth layer : Four nodes with sigmoid function
III. P ROGRESS AND C ONCLUSION
A. Setup
We developed our algorithms on Python3 . And we em-
ployed Tensorflow 1.4,Keras 2.0.9 to build the network and
the loss function . We used CV2 3.3.0 for image processing
and drawing bounding boxes as well as text. For mathematical
computations Numpy 1.2.1 was used . For network model
training NVIDIA NC6 in AWS
B. Dataset
In order to train a convolutional neural network, a lot of
data is required for good accuracy . All the dataset has been
prepared and trained by us.
C. Data Collection
We used OpenCv module of python for image processing,
image is extracted from live video with camera frame by
frame. Images are then converted to grayscale. Adaptive mean
thresholding with binary thresholding is used Mask is applied.
The final image that was saved was a 64x64 grayscaleimage.
D. Data Preprocessing
This step preprocesses the data i.e. normalizing and split-
ting the data into train test. The split ratio is 75
Train 8000
Test 2000
E. CNN Model
Fig 3: Flow Chart Diagram This section describes the archi-
tecture used. The model consists of two CONV BOX followed
by a fully connected layer. One CONV BOX consists of three
layers
Convolution Operation - ¿ Relu - ¿ Max Pooling So, this
operation was carried out twice. Then flattening of the data
is performed and fed into Fully connected ANN.
The plot Fig. 1 shows the steps performs in training the
neural network with a appropriate set of hyperparameter with
proper tuning .

R ESULT

Fig 4: Flow Chart Diagram shows the validation loss for the
training set

Fig 5: Flow Chart Diagram shows the validation loss for the
test set
C ONCLUSION AND F UTURE W ORK
As can be observed from the graph, the accuracy of our
model is very good and hence it will work satisfactorily for
four control based systems .So for the future work we are
planning to develop a multiple control based game and control
it with gestures instead of keyboard.
R EFERENCES
[1] https://www.youtube.com/watch?v=FTr3n7uBIuE
[2] http://cs231n.github.io/convolutional-networks/
[3] https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/
[4] http://colah.github.io/posts/2014-07-Conv-Nets-Modular/
[5] http://andrew.gibiansky.com/blog/machine-learning/convolutional-
neural-networks/