2023 IEEE 14th Control and System Graduate Research Colloquium (ICSGRC), 5 Aug 2023, Shah Alam, Selangor,

Malaysia

Hand Gesture Recognition based on Convolution

Neural Network (CNN) and Support Vector
Machine (SVM)
Muhammad Afiq Abdull Razak Farah Yasmin Abdul Rahman Roslina Mohamad
2023 IEEE 14th Control and System Graduate Research Colloquium (ICSGRC) | 979-8-3503-4623-7/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICSGRC57744.2023.10215427

School of Electrical Engineering
College of Engineering
Universiti Teknologi MARA
Shah Alam, Selangor, Malaysia
afiqre1998@gmail.com
Wireless High Speed Network Research
Group (WHiSNet)
School of Electrical Engineering
College of Engineering
Universiti Teknologi MARA
Shah Alam, Selangor, Malaysia
farahy@uitm.edu.my*
Wireless High Speed Network Research
Group (WHiSNet)
School of Electrical Engineering
College of Engineering
Universiti Teknologi MARA
Shah Alam, Selangor, Malaysia
roslina780@uitm.edu.my

Shahrani Shahbuddin Yuslinda Wati Mohamad Yusof Saiful Izwan Suliman

School of Electrical Engineering School of Electrical Engineering School of Electrical Engineering
College of Engineering College of Engineering College of Engineering
Universiti Teknologi MARA Universiti Teknologi MARA Universiti Teknologi MARA
Shah Alam, Selangor, Malaysia Shah Alam, Selangor, Malaysia Shah Alam, Selangor, Malaysia
shahranis@uitm.edu.my yuslinda@uitm.edu.my saifulizwan@uitm.edu.my

Abstract— Gestural communication is a type of nonverbal
communication in which visible body gestures are utilised to
communicate vital messages, either in place of speech or in
conjunction with it. The problem of gesture division is
presented as a first step toward visual hand gesture
recognition, i.e., the detection, analysis, and recognition of
gestures through real-time hand sequences. Visual hand
recognition and motion tracking are quite challenging to solve
due to their inconvenient nature. This research seeks to
address the issue by determining which classification
technique, Convolutional Neural Network (CNN) or Support
Vector Machine (SVM), is superior in recognising hand
motions. The hand-skeletal was used as the features to
represent the hand gestures. Both classification methods
utilised the same sample dataset and camera input to achieve a
fair comparison. Then, the performance in terms of accuracy
and processing time being analysed. The results indicate that
the CNN excels in recognising hand gestures with an accuracy
of 97.78% compared to the SVM with 96.30%. In terms of
processing time to train/process the datasets, SVM has the
upper hand by taking 5 minutes and 16 seconds. Meanwhile
the CNN used 8 minutes and 24 seconds.
Keywords—Hand Gesture Recognition, Hand Skeletal, CNN,
SVM, Accuracy, Processing Time
I. INTRODUCTION
Hand gesture recognition is critical to the effectiveness of
human-computer interaction (HCI) technology, which is
used as a helpful interface in a variety of challenging
scenarios [1]. The system allows for nonverbal
communication that is natural, inventive, and modern. It can
also be used in a variety of settings [3]. As a result, fostering
natural HCI is critical to bridging the gap between humans
and computers [2]. Further, HCI technology can be used to
create a smart environment [4].
Hand gestures have become a popular way of
communicating simple thoughts, which are then translated
into events by a gesture detection system [3]. However,
analysing the complete number of features takes a long time
[1]. Every level of development aimed to improve the
computer's intelligence and make it easier for people to
communicate with computers in more intricate ways [4].
Technology needs to be able to recognize, classify, and
interpret many simple hand gestures and use them in a wide
range of situations [3].
The main purpose of this research is to recognize nine
(9) hand gestures which consist of “Call", "Fist", "Live
Long", "Okay", "Peace", "Rock", "Stop", "Thumbs Up" and
"Thumbs Down" hand gestures. The recognition system was
developed using the Python programming language. The
feature extraction technique namely hand skeletal-based
technique was used to represent the hand gestures. Then,
CNN and SVM were used as recognition algorithms. Finally,
we analyse the recognition techniques based on their
accuracy and processing time.
II. LITERATURE REVIEW
Hand gesture recognition has become one of research
topics that gain attentions of many researchers including
works presented in [1] – [10]. There were many approaches
being introduced, but the one that we interested to
investigate is work done by Md Abdur Rahim et al [1]. They
proposed Skeleton Distance Measurement (SDM) as feature
extraction technique to represent hand gestures. This
approach was very straight forward where prominent points
on hand being detected and distance among those points
being calculated and became the input of the classifier. CNN
is one of the classifiers that gave high recognition rate in
hand gesture recognition research. For example, research
presented by Md Rashedul Islam et al [9]. Their work able
to obtain classification accuracy at 98.41%. Besides CNN,
SVM also able to give high accuracy in recognising hand
gestures. Research done by Chin-Pan Huang et al [10] used
(SVM) algorithm as their recognition algorithm and gain
recognition rate of 97.51%.
Based on our reviews, both CNN and SVM able to give
very good accuracy which is more than 90% in hand gesture
recognition studies. Therefore, we chose to used both

979-8-3503-4623-7/23/$31.00 ©2023 IEEE 123

Authorized licensed use limited to: Mukesh Patel School of Technology & Engineering. Downloaded on September 21,2023 at 17:15:42 UTC from IEEE Xplore. Restrictions apply.
 2023 IEEE 14th Control and System Graduate Research Colloquium (ICSGRC), 5 Aug 2023, Shah Alam, Selangor, Malaysia
techniques and investigate their performance in terms of
accuracy and processing time.
III. METHODOLOGY
Figure 1: Flowchart of the Project
Figure 1 above shows the flowchart of this project. It
consists of data acquisition, feature extraction, classification
and lastly, analysis of the findings based on recognition
results.
A. Data Acquisition
Figure 2 shows examples of dataset that consist images
of hand gesture used in this research. The dataset was taken
from our own collections where the images were obtained
using a digital camera with a resolution of 0.9MP with a
224x224 dimension of frame size. They were 9 types of
gestures being studied in this project. They were classified
as "Call", "Fist", "Live Long", "Okay", "Peace", "Rock",
"Stop", "Thumbs Up" and "Thumbs Down" Gestures.
Figure 2: Dataset of Hand Gesture
124
Authorized licensed use limited to: Mukesh Patel School of Technology & Engineering. Downloaded on September 21,2023 at 17:15:42 UTC from IEEE Xplore. Restrictions apply.
During training and validation process, total 2700
images were used. Each gesture has 300 photo samples.
Meanwhile during testing, the images taken in real time
where 5 different people doing all 9 gestures were being
used.
B. Feature Extraction Technique
Figure 3: Flow of Data Extraction
Figure 3 above shows the flow of data extraction in this
project. The key points detection that was then represented
the hand skeletal was utilized by importing the MediaPipe
Library via the Python programming language. The
MediaPipe Library has a pre-installed programme of
skeletal-based hand detection which can be used to detect
each coordinate of hand key point fingertip.
Figure 4: Skeletal-based detection of the hand
Figure 4 shows an example of skeletal-based hand key
points detection. This hand landmark model performed
precise key point localization of 21 points on each finger.
Then, the data from those 21-key points of localization
fingertip were stored into a CSV file by using the Pandas
Library via Python. Pandas is a Python library that is used
for data analysis. The finished CSV file that contains all the
data extracted can then be ready to be trained into both
classification algorithms, which in this research, the CNN
and SVM.
C. Classification Algorithm
1) CNN
Figure 5: The creation of CNN model
Figure 5 above shows the flow of the CNN algorithm
used in this project. The data extracted via the skeletal-based
 2023 IEEE 14th Control and System Graduate Research Colloquium (ICSGRC), 5 Aug 2023, Shah Alam, Selangor, Malaysia

hand detection in the CSV file was then be used to train the
CNN model algorithm. This project utilized the CNN
training algorithm with the import of the TensorFlow
library, which contains the preinstalled programme of the
CNN architecture that can be found in the Python
programming language. The data from the CSV file were
used to train the CNN model.
gestures. Nonetheless, the recognition rate is still good,
which is more than 80%. Next, the "Call" and "Fist gestures
has the same accuracy rate of the 94%. The overall
accuracy of the CNN algorithm is 97.78%.
Table 1: Confusion Matrix for CNN technique

The model consists of a linear stack of layers. This

project used Rectified Linear Unit (ReLU) as the function
activation to create the input layer. The input layer were
created with 512 nodes (neurons) input nodes. Next, the
output layer consists of 9 nodes, as the project wants to
classify the given 9 gestures. All the 512 input nodes were
fully connected to the output layer. A SoftMax function
activation was used to classify the output into 9 distinct
units. After the CNN model successfully created, the model
was loaded into the recognition and tested using real-time
images.

2) SVM

Table 2: Confusion Matrix for SVM technique

Figure 6: Support Vector Classifier with RBF kernel

Figure 6 shows the flow of SVM Algorithm in this

research. The input data were extracted via the skeletal-
based hand detection in the CSV file. This project used the
SVM training algorithm with the import of the SciKit-learn
library, which contains the preinstalled programme of the
SVM architecture, which can be found via the Python
programming language. RBF kernel were used to separate
the SVM hyperplane. After the SVM model successfully
being trained, real-time images taken from digital camera
were used to test the accuracy of the developed recognition
system.
IV. RESULT AND DISCUSIION
Meanwhile, table 2 shows the confusion matrix for
This chapter describes the results obtained from the recognition of hand gestures using SVM technique. SVM
Hand Gesture Recognition using CNN and SVM algorithm is able classify perfectly with 100% recognition
algorithms. rate on 7 hand gestures, which are the "Okay", "Peace",
"Thumbs Down", "Call", "Stop", "Rock" and "Fist"
Table 1 shows the confusion matrix of hand gesture gestures. The "Live Long" gesture has the lowest accuracy
recognition using CNN technique. The confusion matrix of all other hand gestures by the SVM algorithm with
consists of 9 classes of hand gesture. Each gesture was 74.67% because it seems to be misjudged as the "Stop". The
tested 150 times and recognition results were recorded in a reason of this poor performance is all fingers were kept
confusion matrix. From the Table 1, it shows that the CNN straight thus the skeletal shape profile of those hand gestures
algorithm is able to achieve 100% accuracy in 7 hand were almost similar. This made the SVM difficult to set the
gestures, which were the "Okay", "Peace", "Thumbs Up", hyperplane boundary between those 2 gestures. Further, the
"Thumbs Down", "Rock", "Live Long" and "Fist" gestures. "Thumbs Up" gesture has the accuracy of the 92%. Same
The "Stop" gesture has the lowest accuracy among other recognition performance with 92% for "Call" hand gesture.
hand gestures classified by the CNN algorithm with 86% However, the overall accuracy of the SVM algorithm is
because it seems to be misjudged as the "Live long" and 96.30%.
"Rock" gesture. This caused by the similarity of skeletal
profiles where most of the fingers were kept straight in all 3

125
Authorized licensed use limited to: Mukesh Patel School of Technology & Engineering. Downloaded on September 21,2023 at 17:15:42 UTC from IEEE Xplore. Restrictions apply.
 2023 IEEE 14th Control and System Graduate Research Colloquium (ICSGRC), 5 Aug 2023, Shah Alam, Selangor, Malaysia
Figure 7: Performance of each Hand Gesture Class based on CNN and
SVM
Figure 7 shows the bar graph of accuracy for each hand
gesture versus classification algorithm. From the Figure 7, it
shows that there are 5 hand gestures that achieved 100%
recognition rate, which are the "Fist", "Rock", "Thumbs
Down", "Peace", "Stop" and "Okay" gestures. Both
algorithms have trouble recognising the "Live Long"
gestures as the gestures are sometimes misinterpreted as
"Stop" gestures. Based on Figure 7 also shows that CNN
successfully recognizing all the studied 9 hand gestures with
accuracy above 85%. In terms of overall accuracy, both
classification algorithms have achieved the satisfactory
recognition rate. CNN excels in recognising hand gestures
with an accuracy of 97.78% compared to the SVM with
96.30%.
Figure 8: Processing Time for Hand Gesture Recognition based on CNN
and SVM
Next, Figure 8 shows the processing time of hand
gesture recognition using CNN and SVM. From Figure 8, it
shows that SVM need less processing time compared to
CNN. This is due to SVM doesn’t require to validate the
trained dataset. The hyperplane able to be separated by its
class in 5 minutes and 8 seconds. Meanwhile the
Convolutional Neural Network (CNN) used 8 minutes and
23 seconds to train and validate the dataset.
126
Authorized licensed use limited to: Mukesh Patel School of Technology & Engineering. Downloaded on September 21,2023 at 17:15:42 UTC from IEEE Xplore. Restrictions apply.
V. CONCLUSION AND RECOMMENDATION
In conclusion, this research successfully developed a
hand recognition system using CNN and SVM. Skeletal-
based hand detection method is suitable to be used as feature
extraction technique to represent a hand gesture. Both CNN
and SVM able to recognized all 9 hand gestures. CNN has
better performance in terms of accuracy. However, in terms
of processing time, SVM works much faster than CNN.
REFERENCES
[1] M. A. Rahim, A. S. M. Miah, A. Sayeed and J. Shin, "Hand Gesture
Recognition Based on Optimal Segmentation in Human-Computer
Interaction," 2020 3rd IEEE International Conference on Knowledge
Innovation and Invention (ICKII), 2020, pp. 163-166, doi:
10.1109/ICKII50300.2020.9318870.
[2] O. Köpüklü, T. Ledwon, Y. Rong, N. Kose and G. Rigoll,
"DriverMHG: A Multi-Modal Dataset for Dynamic Recognition of
Driver Micro Hand Gestures and a Real-Time Recognition
Framework," 2020 15th IEEE International Conference on Automatic
Face and Gesture Recognition (FG 2020), 2020, pp. 77-84, doi:
10.1109/FG47880.2020.00041.
[3] M. Panwar and P. Singh Mehra, "Hand gesture recognition for human
computer interaction," 2011 International Conference on Image
Information Processing, 2011, pp. 1-7, doi:
10.1109/ICIIP.2011.6108940.
[4] S. Veluchamy, L. R. Karlmarx and J. J. Sudha, "Vision based
gesturally controllable human computer interaction system," 2015
International Conference on Smart Technologies and Management for
Computing, Communication, Controls, Energy and Materials
(ICSTM), 2015, pp. 8-15, doi: 10.1109/ICSTM.2015.7225383.
[5] Lae-Kyoung Lee, Su-Yong An and Se-Young Oh, "Robust fingertip
extraction with improved skin color segmentation for finger gesture
recognition in Human-robot interaction," 2012 IEEE Congress on
Evolutionary Computation, 2012, pp. 1-7, doi:
10.1109/CEC.2012.6256140.
[6] P. Choudhary and S. N. Tazi, "An Adaptive System of Yogic Gesture
Recognition for Human Computer Interaction," 2020 IEEE 15th
International Conference on Industrial and Information Systems
(ICIIS), 2020, pp. 399-402, doi: 10.1109/ICIIS51140.2020.9342678.
[7] Y. F. A. Gaus and F. Wong, "Hidden Markov Model-Based Gesture
Recognition with Overlapping Hand-Head/Hand-Hand Estimated
Using Kalman Filter," 2012 Third International Conference on
Intelligent Systems Modelling and Simulation, 2012, pp. 262-267,
doi: 10.1109/ISMS.2012.67.
[8] Z. Yang, Y. Li, W. Chen and Y. Zheng, "Dynamic hand gesture
recognition using hidden Markov models," 2012 7th International
Conference on Computer Science & Education (ICCSE), 2012, pp.
360-365, doi: 10.1109/ICCSE.2012.6295092.
[9] M. R. Islam, U. K. Mitu, R. A. Bhuiyan and J. Shin, "Hand Gesture
Feature Extraction Using Deep Convolutional Neural Network for
Recognizing American Sign Language," 2018 4th International
Conference on Frontiers of Signal Processing (ICFSP), 2018, pp. 115-
119, doi: 10.1109/ICFSP.2018.8552044.
[10] C. -P. Huang, C. -H. Hsieh, K. -T. Lai and W. -Y. Huang, "Human
Action Recognition Using Histogram of Oriented Gradient of Motion
History Image," 2011 First International Conference on
Instrumentation, Measurement, Computer, Communication and
Control, 2011, pp. 353-356, doi: 10.1109/IMCCC.2011.9