IJARSCT

ISSN (Online) 2581-9429

International Journal of Advanced Research in Science, Communication and Technology (IJARSCT)

<May > 2022

Impact Factor:

.
Hand Gesture Mouse Recognition implementation to
replace hardware mouse using AI
Gaurav Munjewar1 Geetesh Kongre2 Prof, Sunita P. Deshmukh3
Department of E & Tc, NBN SINHGAD School of Engineering, Pune,
India

Abstract:
Natural user interfaces have become increasingly important in today's world, because to advances in ubiquitous
computing. The presence of computers and the use of human-computer interaction tools in our society will undoubtedly
have a positive impact on our civilizations. Whether it was back in the day when technology was less advanced or today
when technology has advanced so much that we spend the majority of our time communicating, playing, doing our jobs
with machines, and so on, human beings have used and continue to use a wide range of gestures to communicate and
interact with one another. Human gestures are a type of nonverbal communication that can be used to convey
information. We can use this vision based recognized gestures to control multimedia applications (like Windows Media
Player, Windows Picture Manager, VLC Player etc.) running on computer using different gestural commands.

I. INTRODUCTION

The virtual mouse establishes a virtual connection between the user and the machine without the use of any hardware.
This gesture recognition system can capture and track the fingertips of a person wearing a color cap with a webcam,
and the system detects the hand's color and movements and moves the cursor along with it.
The following describes the overall objectives of this project:
 To design to operate with the help of a webcam. The Virtual Mouse application will be operational with the help of a
webcam, as the webcam are responsible to capture the images in real time. The application would not work if there are
no
webcam detected.
 To design a virtual input that can operate on all surfaces. The Virtual Mouse application will be operational on all
surface and indoor environment, as long the users are facing the webcam while doing the motion gesture.
 To program the camera to continuously capturing the images, which the images will be analyzed, by using various
image processing techniques. As stated above, the Virtual Mouse application will be continuously capturing the images
in real time, where the images will be undergoing a series of process, this includes HSV conversion, Binary Image
conversion, salt and pepper noise filtering, and more.
 To transform a hand gesture or motion into mouse input for a certain screen position. The Virtual Mouse application
will be programmed to detect the position of the defined colors where it will be set as the position of the mouse pointers.
Furthermore, a combination of different colors may result in triggering different types of mouse events, such as the
right/left clicks, scroll up/down, and more.

II. BRIEF LITERATURE SURVEY

The current system is comprised of a generic mouse and trackpad monitor control system, as well as the absence of a
hand gesture control system. The use of a hand gesture to access the monitor screen from a distance is not possible.
Even though it is primarily attempting to implement, the scope is simply limited in the virtual mouse field.
The existing virtual mouse control system consists of simple mouse operations using a hand recognition system, in
which we can control the mouse pointer and left click and so on. The use of hand recognition in the future will not be
used. Even though there are a variety of systems for hand recognition, the system they used is static hand recognition,
which is simply recognition of the shape made by the hand and the definition of action for each shape made, which is
limited to a few defined actions and causes a lot of confusion. As technology advances, there are more and more
alternatives to using a mouse.
The following are some of the techniques that were employed.

Copyright to IJARSCT
www.ijarsct.co.in
DOI 10.48175/IJARSCT-XXXX
1
 IJARSCT
ISSN (Online) 2581-9429

International Journal of Advanced Research in Science, Communication and Technology (IJARSCT)

<May > 2022

Impact Factor:
A) Head Control: A special sensor (or built-in webcam) can track head movement to move the mouse pointer around
on the screen. The dwell delay function of the software is frequently employed in the absence of a mouse button.
Clicking can also be accomplished with a well-placed switch.

B) Eye Control: The cost of modern eye gaze systems is decreasing. These enable users to move the pointer on the
screen solely by moving their eyes. Instead of mouse buttons, a dwell delay feature, blinks, or a switch are used.
The Tobi PCEye Go is a peripheral eye tracker that lets you use your eyes to control your computer as if you
were using a mouse.
C) Touch Screens: Touch screens, which were once seen as a niche technology used primarily in special education
schools, have now become mainstream. Following the success of smartphones and tablets, touch-enabled
Windows laptops and all-in-one desktops are becoming more common. Although this is a welcome new
technology, the widespread use of touch screens has resulted in a new set of touch accessibility issues. However,
each of the methods below has its own set of disadvantages. The use of the head or eyes to control the cursor
regularly can be hazardous to one's health. This can lead to a number of problems with health. When using a
touch screen, the user must maintain their focus on the screen at all times, which can cause drowsiness. By
D) comparing the following techniques, we hope to create a new project that will not harm the user's health.

III. IMPLEMENTATION OF HARDWARE

For the implementation of hand gesture recognition (HGR) system or multimedia interaction we have built our own data
set. We have recorded each video stream of duration time approximately 10 seconds at the rate of 30 frames per seconds
and at the resolution of 1280×720 using digital camera of 8 megapixels. The experiments were carried out in three different
sessions. The images (i.e. frames) taken from the recorded video streams at various distances and places are used to classify
these sessions. Each session consists of 300 images of 6 different classes where each class having 50 images. Some samples
of images used in different sessions are shown in the Figs. 2, 3 and 4 based on which we will calculate the hand gesture
recognition accuracies. The whole system is designed using image processing and computer vision techniques implemented
in Matlab-2013under Windows 8 operating system. The hardware required for processing the hand motion recognition
system was a 64-bit PC with a 2.40GHz processor and 2 GB RAM.

1. Capturing real time video using Web-Camera: We will need a sensor to detect the user's hand movements in order for
the system to work. As a sensor, the computer's webcam is used. The webcam records real-time video at a fixed frame rate
and resolution determined by the camera's hardware. If necessary, the system's frame rate and resolution can be changed.
2.Converting the video captured into HSV format: The video has also been converted into HSV (hue, saturation, meaning,
also called HSB), an alternative representation of the RGB color model created by computer graphics researchers to better
reflect the perception of colored characteristics by human vision.
3) Each image frame is processed separately: Following the capture of the video, it goes through a brief pre-processing
stage before being processed one frame at a time.
4) Conversion of each frame to a greyscale image: A gray scale image has a lower computational complexity than a
colored image. It also aids in faster color calibration without the use of external noise. All the necessary operations were
carried out after the image was converted to gray scale.
5) Calibrate the color ranges: The device enters the calibration mode after the above steps, which assigns each color
according to the HSV rule to its color hue, saturation or value values. Every color already has its predetermined values.
For accurate color detection, the user can adjust the ranges. In the diagram below you can clearly see the variety of values
used to detect every color cap
6) Calculate the image's centroid by locating the image's region. To guide the mouse pointer, the user must first choose a
point whose coordinates can be sent to the cursor. The device can monitor cursor movement using these coordinates. As
the object travels around the frame, these coordinates change over time.
7) Tracking the mouse pointer. After determining the coordinates, the mouse driver is accessed and the coordinates are
sent to the cursor. The cursor positions itself in the required position using these coordinates. As a result, the mouse moves
proportionally across the screen as the user moves his hands across the camera's field of view.

8) Simulating the mouse actions. To create the control actions in simulation mode, the user must make hand gestures.
Because color pointers are used, the computation time is lowered.

Copyright to IJARSCT
www.ijarsct.co.in
DOI 10.48175/IJARSCT-XXXX
2
 IJARSCT
ISSN (Online) 2581-9429

International Journal of Advanced Research in Science, Communication and Technology (IJARSCT)

<May > 2022

Impact Factor:

IV. IMPLEMENTATION OF SOFTWARE

We conceived and designed a system for vision-based hand gesture identification that included several stages that we
explained using an algorithm. Flowchart of hand gesture recognition is the functioning flowchart of the gesture
recognition system.’

ALGORITHM:

1. Extract a frame (i.e. hand image) from recorded video stream.

2. Extracted frame is transformed from RGB color space to color space model and then hand is detected in the image
using skin color-based detection techniques.
3. After detection of hand, we have converted the image into black & white i.e., marked the skin pixels as white and on
skin pixels (i.e. background) as black and then we have applied some preprocessing techniques like image filling,
morphological erosion using 15×15 structuring elements etc. to increase the quality of image and to remove some
noise.
4. For the feature extraction centroid, equivalent diameter, area, perimeter and orientation of detected objects is found
out in the frame. With the help of centroid and diameter a circle is drawn same as background color pixels.
5. Gesture is recognized by counting the number of white objects in the image and orientation of image. Finally
acommand is passed to the applications running on the computer corresponding to the recognized gesture.

Copyright to IJARSCT
www.ijarsct.co.in
DOI 10.48175/IJARSCT-XXXX
3
 IJARSCT
ISSN (Online) 2581-9429

International Journal of Advanced Research in Science, Communication and Technology (IJARSCT)

<May > 2022

Impact Factor:

V. RESULTS

After repeating the practical testing, it was discovered that the system performs exceptionally well under adequate
illumination and in a plain background environment devoid of any skin-like items. The method is not particularly robust
because it finds it difficult to distinguish the hand against a complicated background.

Fig. A Fig. B Fig. C Fig. D

The system's performance is also influenced by the threshold value used to determine the radius using various feasible
approximations. Nonetheless, the system is slightly more responsive than other systems that have been built previously
because it does not require a training period for gesture recognition.

VI. CONCLUSION AND DISCUSSION

The AI virtual mouse system's main goal is to control mouse cursor functionalities with hand gestures rather than
utilizing a hardware mouse. The proposed system can be achieved by using a webcam or a built-in camera which detects
the hand gestures and hand tip and processes these frames to perform the particular mouse functions.
The proposed AI virtual mouse has various flaws, such as a slight loss of accuracy when using the right-click mouse
function, and the model has some difficulty selecting text by clicking and dragging. These are some of the suggested AI
virtual mouse system's shortcomings, which will be addressed in future research.

VII. ACKNOWLEDGEMENT

I am thankful to Prof. Sunita P. Deshmukh Project Guide, for her co-operation and suggestions. She has made
valuable suggestions and gave guidance throughout for completing my project work. I am also thankful to Prof.
S.Y. Tamboli, project coordinator for his help and support.
I wish to express deep heartfelt thanks to Head of Department, Dr. Makarand M. Jadhav, for giving valuable
guidance and direction to my project work. He patiently provided the encouragement and motivation. His
comments and precious suggestions made the entire journey to the final accomplishment of this work really
pleasant.
I am also thankful to Dr. Shivprasad. P. Patil, Principal, NBN Sinhgad School of Engineering, Pune for
providing excellent facilities in the college to carry out this project work.
I wish to thank all the faculty and staff members of department for their sincere co-operation, kind blessings and
warm well-wishes.

REFERENCES

[1] S. S. Rautaray and A. Agrawal, Vision Based Hand Gesture Recognition for Human Computer Interaction: A
survey, Springer Transactional Artificial Intelligence Review, pp. 1–54, (2012).
[2] P. Payeur, C. Pasca, A. Cretu and E. M. Petriu, Intelligent Haptic Sensor System for Robotic Manipulation,
Copyright to IJARSCT
www.ijarsct.co.in
DOI 10.48175/IJARSCT-XXXX
4
 IJARSCT
ISSN (Online) 2581-9429

International Journal of Advanced Research in Science, Communication and Technology (IJARSCT)

<May > 2022

Impact Factor:
IEEE Transaction on Instrumentation and Measurement, vol. 54(4), pp. 1583–1592, (2005).
[3] S. Meena, A Study on Hand Gesture Recognition Technique, Master Thesis, Department of Electronics
and Communication Engineering, National Institute of Technology, India, (2011).

[4] M. M. Hasan and P. K. Mishra, Hand Gesture Modelling and Recognition using Geometric Features: A
Review, Canadian Journal on Image Processing and Computer Vision, vol. 3(1), pp. 12–26, (2012).
[5] B. A. Myers, A Brief History of Human Computer Interaction Technology, ACM Interactions, vol. 5(2), pp. 44–54
A Fast Algorithm for Vision-Based Hand Gesture Recognition For Robot Control, IEEE Signal Processing and
Communications Applications, pp. 1–4, (2006).
[6] Z. Xu, et al., Hand Gesture Recognition and Virtual Game Control Based on 3D Accelerometer and EMG Sensors,
(2009).
[7] S. Mitra and T. Acharya, Gesture Recognition: A Survey, IEEE Transactions on Systems, Man and
Cybernetics, Part C: Applications and Reviews, vol. 37(3), pp. 311–324, (2007).
[8] N. X. Tran, Wireless Data Glove for Gesture-Based Robotic Control, 13thInternational Conference on HCI,
vol. 5611, pp. 271–280, (2009).
[9] J. M. Rehg and T. Kanade, Visual Tracking of High DOF Articulated Structures: An Application to Human
Hand Tracking, 3rdEuropeanConference on Computer Vision, pp. 35–46, (1994).

Copyright to IJARSCT
www.ijarsct.co.in
DOI 10.48175/IJARSCT-XXXX
5
 IJARSCT
ISSN (Online) 2581-9429

International Journal of Advanced Research in Science, Communication and Technology (IJARSCT)

<May > 2022

Impact Factor:

BIOGRAPHY

Author 1 Gaurav Munjewar (PRN: - 71804123C) NBN Sinhgad School of

Engineering, Pune.

Author 2 Geetesh Kongre (PRN: - 71726431K) NBN Sinhgad School

of Engineering, Pune.

Project Guide Prof. Sunita. P. Deshmukh is working as a Professor at NBN Sinhgad School of
Engineering, Pune. Her area of interest is VLSI and Embedded Systems.

Copyright to IJARSCT
www.ijarsct.co.in
DOI 10.48175/IJARSCT-XXXX
6