HAND GESTURE

BASED
VIRTUAL MOUSE

Under the guidance of Mr.K. Vikram Reddy

1608-17-733-071 Dudipala Rushikethan Reddy
1608-17-733-086 Voruganti Sai Krishna
1608-17-733-111 Falgun Reddy
ABSTRACT
➢ As the technologies are developing day by day the devices becoming
compact in size.
➢ The proposal is to development of a Virtual Mouse using Gesture
Recognition.
➢ The aim is to control mouse cursor functions using only a simple camera
instead of a traditional or regular mouse device.
➢ The Virtual Mouse works as a medium of the user and the machine only
using a camera. It helps the user to interact with a machine without any
mechanical or physical devices and control mouse functions.
➢ In this gesture recognition system, it is very possible to capture & track
the fingertip of hand with a webcam or built-in cam which is bearing a
color cap or color sticky note paper and the system track the color and
movement of the hand & move cursor with it.
BASE PAPER EXPLANATION AND
OBSERVATIONS
➢ Basically there could be two possible gesture recognitions for HCI, one is
hardware based & another one is computer vision based.
➢ One of the early hardware-based systems was proposed by Quam (1990)
in which, the user had to wear a bulky DataGlove to use the system .
➢ Though this method gives a high accuracy control, it is very difficult to use
as some gesture is not meant for everyone and also very impractical for
mass users in the everyday world.
➢ Meanwhile, the vision-based hand gesture recognition is also two types:
marker-based & non-marker based.
BASE PAPER EXPLANATION AND
OBSERVATIONS
➢ Nonmarker based recognition generally is not as accurate as the marker
based recognition.

➢ And the maker based recognition has better accuracy compared to the
other gesture recognition systems, though the user has to use a simple
color cap on the fingertip.

➢ But this way is lighter and almost nothing in comparison with the
DataGlove of the hardware-based system.
PROBLEM DEFINITION
➢ We use coloured tips for detection which are captured by webcam.
➢ Controlling the mouse functions using hand gestures. The main functions
are mouse movement, left button click, right button click, double click and
scrolling up or down.
➢ Implemented in Python programming language using the Computer
Vision based library OpenCV.
➢ This system has the potential to replace the typical mouse.
PROPOSED EXTENSION TO BASE PAPER
➢ Lots of study has been done on sensor based approaches like gloves,
wires etc. but due to disadvantage of wear it continuously is not possible,
therefore further work is concentrated on Image based approaches

➢ Training accuracy and validation accuracies for this CNN architecture are
slightly better than the previously proposed SLR models. A less amount of
training and validation loss is observed with the proposed CNN
architecture.
PROPOSED EXTENSION TO BASE PAPER
➢ This system uses CNN algorithm which automatically detects the important
features without any human supervision.

➢ CNN is also computationally efficient.

➢ CNN are hugely popular is because of their architecture — the best thing is
there is no need of feature extraction.

➢ Feature extraction is a type of dimensionality reduction where a large number

of pixels of the image are efficiently represented in such a way that interesting
parts of the image are captured effectively.
METHODOLOGY
METHODOLOGY
➢ Camera: The system works on the frames captured by the webcam on
the computer machine or built-in camera on a laptop. By creating the
video capture object, the system will capture video using webcam in real-
time. In order to use a single camera with this system the device index
would be “0”. To add additional camera device index would be 1, 2 and so
on. This camera will capture frame by frame and will pass it to the system.
➢ Capturing: By using an infinite loop the webcam captures each and every
frame till the program termination. The frames from the real-time video
are processed from BGR to HSV color space.
METHODOLOGY
➢ Color Detection & Masking: In this proposed system, color detection is
done by detecting color pixels of fingertips with color caps from the
frames that were captured by the webcam. This is the initial and
fundamental step of the proposed system. The outcome of this step will
be a grayscale image, where the intensity of the pixels differs the color
cap from the rest of the frame and the color cap area will be highlighted.
Then rectangle bounding boxes (masks) will be created around the color
cap and the color cap will be tracked. The gesture will be detected from
the tracking of these color caps.
PLANNING FOR FUTURE WORK
REFERENCES
➢ Itay Lieder, Tom Hope, and Yehezkel S. Resheff, Learning TensorFlow : A
Guide to Building Deep Learning Systems.

➢ C.-Chiung Hsieh, D.-Hua Liou, & D. Lee, , “A real time hand gesture
recognition system using motion history image,” Proc. IEEE Int'l Conf.
Signal Processing Systems (ICSPS).

➢ https://towardsdatascience.com/sign-language-recognition-using-deep-le
arning-6549268c60bd

➢ Quam, “Gesture recognition with a Data Glove”.

Thank you