VIRTUAL MOUSE USING HAND GESTURES

A PROJECT REPORT
Submitted
in partial fulfillment for the award of the degree of
Bachelor of Technology
IN
COMPUTER SCIENCE & ENGINEERING

Name of supervisor Name of student

Ruchi rai Apeksha Tomar
Hrithik Sharma

DEPARTMENT OF COMPUTER SCIENCE ENGINEERING

SHRI RAM GROUP OF COLLEGES, MUZAFFARNAGAR, U.P.
AFFILIATED TO DR. A.P.J. ABDUL KALAM TECHNICAL UNIVERSITY, LUCKNOW, U.P.

2022
 DECLARATION BY THE CANDIDATE

I hereby declare that the Project entitled “VIRTUAL MOUSE USING HAND
GESTURES” submitted for the B.Tech. in Computer Science & Engineering degree is
my original work and the project has not formed the basis for the award of any other
degree, diploma, fellowship or any other similar titles.
I declare that I have not willfully lifted up some other’s work, para, text, data, results,
etc. reported in the journals, books, magazines, reports, dissertations, etc. or available
at websites.

Date: Signature:
Place: Apeksha Tomar, Hrithik Sharma
1869510013,1869510024

ii
 CERTIFICATE

This is to certify that the Project thesis, entitled VIRTUAL MOUSE USING HAND
GESTURES submitted by MS APEKSHA TOMAR and MR HRITHIK SHARMA
embodies the findings of her original project/research work carried out under my
supervision and it fulfills all the conditions prescribed by Dr. A.P.J. Abdul Kalam
Technical University , Lucknow for the award of Bachelor of Technology Degree in
Department of Computer Science Engineering the best of my/our knowledge, the
matter embodied in this Project/ thesis has not been submitted elsewhere for the award
of any other degree or diploma .

Signature and Name of Supervisor: RUCHI RAI

Designation: Asst. Professor
Department: Computer Science Engineering

Signature and Name of Head of Department: Mr. Pawan Kumar Goel

Designation: Associate Professor and Head of Department
Department: Computer Science Engineering

Signature of External Examiner:

Date :

iii
 ACKNOWLEDGEMENT

It gives us a great sense of pleasure to present the report of the B. Tech Project
undertaken during B. Tech. Final Year. We owe special debt of gratitude to our
Supervisor Asst. Prof. Ruchi Rai , Department of Computer Science &
Engineering, Shri Ram Group of Colleges , Muzaffarnagar for his constant support
and guidance throughout the course of our work. His sincerity, thoroughness and
perseverance have been a constant source of inspiration for us. It is only his
cognizant efforts that our endeavours have seen light of the day.

We also take the opportunity to acknowledge the contribution of Prof. Pawan

Kumar Goel , (Head Department of Computer Science & Engineering) for his
full support and assistance during the development of the project.

We also do not like to miss the opportunity to acknowledge the contribution of all
faculty members of the department for their kind assistance and cooperation during
for their contribution in the completion of the project.

Name : APEKSHA TOMAR

Roll No.: 1869510013
Date :
Signature:

Name : HRITHIK SHARMA

Roll No.: 1869510024
Date :
Signature:

iv
 ABSTRACT

This project promotes an approach for the Human Computer Interaction (HCI)
where cursor movement can be controlled using a real-time camera, it is an
alternative to the current methods including manual input of buttons or changing
the positions of a physical computer mouse. Instead, it utilizes a camera and
computer vision technology to control various mouse events and is capable of
performing every task that the physical computer mouse can.

The Virtual Mouse using hand gesture program will constantly acquiring real-time
images where the images will undergone a series of filtration and conversion.
Whenever the process is complete, the program will apply the image processing
technique to obtain the coordinates of the targeted gestures from the converted
frames. After that, it will proceed to compare the existing gestures within the
frames with a list of gestures combinations, where different combinations consists
of different mouse functions. If the current gesture found a match, the program will
execute the mouse function, which will be translated into an actual mouse function
to the users' machine.

Hand gestures are unprompted and also robust transmission mode for Human
computer Interaction (HCI). Keyboard, mouse, joystick or touch screen are some
input device for connection with the computer but they don't provide appropriate
interface whereas, the current system will contain either desktop or laptop interface
in which hand gesture can be done by wearing data gloves or web camera used for
snapping hand image. The first step towards this gesture recognition is hand
capturing and analyzing. Sensors are used in Data-Glove methods for initializing
fingers movement and other sensor will program hand movements. In comparison
the vision based method only needs a camera and hence identifying the actual
interaction between human and computer without using any other devices. The
challenges of this system are constant background, sometimes person and lighting
also. Different procedure and algorithms which are used in this system are
elaborated here along with the recognition techniques. The method of searching a

v
connecting region in the picture with particular specification, being it color or
intensity, where a pattern and algorithm is adjustable is known as segmentation.

With the development technologies in the areas of augmented reality and devices
that we use in our daily life, these devices are becoming compact in the form of
Bluetooth or wireless technologies. This is paper proposes an AI virtual mouse
system that makes use of the hand gestures and hand tip detection for performing
mouse functions in the computer using computer vision. The main objective of the
proposed system is to perform computer mouse cursor functions and scroll
function using a web camera or a built-in camera in the computer instead of using
a traditional mouse device.

Hand gesture and hand tip detection by using computer vision is used as a HCI
with the computer. With the use of the AI virtual mouse system, we can track the
fingertip of the hand gesture by using a built-in camera or web camera and perform
the mouse cursor operations and scrolling function and also move the cursor with
it. While using a wireless or a Bluetooth mouse, some devices such as the mouse,
the dongle to connect to the PC, and also, a battery to power the mouse to operate
are used, but in this paper, the user uses his/her built-in camera or a webcam and
uses his/her hand gestures to control the computer mouse operations.

In the proposed system, the web camera captures and then processes the frames
that have been captured and then recognizes the various hand gestures and hand tip
gestures and then performs the particular mouse function. Python programming
language is used for developing the AI virtual mouse system, and also, OpenCV
which is the library for computer vision is used in the AI virtual mouse system. In
the proposed AI virtual mouse system, the model makes use of the MediaPipe
package for the tracking of the hands and for tracking of the tip of the hands, and
also, Pynput, Autopy, and PyAutoGUI packages were used for moving around the
window screen of the computer for performing functions such as left click, right
click.

vi
LIST OF FIGURES

Figure 1.1: Mechanical mouse, with top cover removed 2

Figure 1.2: Optical Mouse, with top cover removed 6
Figure 2.1: The system overview of Visual Panel 15
Figure 2.2: The system architecture of the mouse-simulation 16
Figure 2.3: Input image using one and two pointers 18
Figure 3.1: Virtual Mouse gesture Recognition Block Diagram 23
Figure 3.2: The comparison between un-filtered and filtered frame 24
Figure 3.3: Volume up 25
Figure 3.4: Left click 26
Figure 3.5: Right click 26
Figure 3.6: No movement of cursor 27
Figure 3.7: Movement of cursor 27
Figure 4.1: Agile method overview 35

ix
LIST OF TABLES

Table 1.2: Advantage and disadvantage of the Optical and Laser Mouse ............. Error!
Bookmark not defined.

Table 4.1: Verification Plan P1 .................................................................................. 46

Table 4.2: Verification Plan P2 ....................................... Error! Bookmark not defined.

Table 4.3: Verification Plan P3 ..................................... Error! Bookmark not defined.8

Table 5.1: Different brightness testing results ........................................................... 52

Table 5.2: gestures conflicts testing results ............................................................... 53

x
LIST OF ABBREVIATIONS

GUI Graphical User Interface

HCI Human Computer Interaction
MHI Motion History Images
IDE Integrated Development Environment
OpenCV Open Source Computer Vision

xi