A

SYNOPSIS
On

“Advancement in Human Interaction: A shift Towards

Device Free Virtual Mouse Control”

Submitted to:
Rashtrasant Tukadoji Maharaj Nagpur University, Nagpur for
Partial Fulfillment of the Degree of

Bachelor of Engineering in
Computer Science and Engineering
Submitted by:
Mr. Naveen Kumar Pathak Ms. Shweta Pardhi
Mr. Praveen Kumar Pathak Mr. Subodh Wathore
Mr. Kunal Rathore

Project Co coordinator
Prof. Aditya Lavahale sir

Department of Computer Science and Engineering

NAAC Accredited with A+ Grade ISO 9001:2015 Certified

Tulsiramji Gaikwad-Patil College of Engineering &

Technology, Nagpur-441108
(An Autonomous Institute affiliated to RTMNU, Nagpur)

Session 2023-24
 Abstract
In the realm of Human-Computer Interaction (HCI), the conventional mouse, although wireless and
Bluetooth versions have emerged, still relies on batteries and dongles. This paper introduces a groundbreaking
solution: a device-independent virtual mouse system utilizing computer vision and hand gestures. By employing
a webcam and sophisticated color segmentation techniques, users can control the computer cursor through
specific gestures, eliminating the need for additional hardware. This paper delves into the algorithms and
methodologies behind hand gesture recognition and color detection, presenting a viable alternative to traditional
touchscreens and physical mice. The proposed system offers an innovative way to interact with computers,
enhancing the user experience and expanding the possibilities of HCI technology.
 Content
Sr. Topic Page No.
No
1. Introduction to Project: Project Title 1

1.1 Introduction to Project 1

1.2 Motivation behind Project topic 1

1.3 Goals & Objective(s) of the work 2

1.4 Implementation 2

1.5 Algorithms and Methods 5

1.6 Experiment Result 6

1.7 Future Scope 6

1.8 Conclusion 6

2. Literature Review 7

3. Applications of Project 8

4. References 9
 1.INTRODUCTION
1.1 Introduction
In the rapidly advancing landscape of technology, devices are becoming increasingly compact, wireless, and
latent, paving the way for innovative approaches in Human-Computer Interaction (HCI). This paper introduces a
groundbreaking concept: the development of a Virtual Mouse using Gesture Recognition, heralding the future of
HCI. The objective is to replace traditional mouse devices with a system controlled solely by a camera, offering
a device-free interaction experience between the user and the machine.

Traditionally, computer interaction relies on physical input devices such as mice and keyboards. Even
wireless devices require power sources and connecting technologies. This proposed Virtual Mouse system
disrupts this paradigm. By utilizing hand gestures captured through a simple camera, users can seamlessly
control mouse cursor functions. A webcam tracks fingertip movement, enabling gestures like right-click,
double-click.This system, implemented in Python using the OpenCV library, demonstrates an interactive and
intuitive way to operate a computer, eliminating the need for physical devices.

The proposed system operates through a fixed integrated laptop camera, minimizing costs. Hand gestures are
captured and tracked using the Red Green Blue (RGB) color space, converted to HSV scheme for system
interpretation. A robust hand gesture segmentation technique is employed, involving pre-processing,
background subtraction, and edge detection. Proper setting of HSV ranges is crucial for accurate object
detection and tracking. This system mimics all functionalities of a traditional mouse, including left-click, right-
click, and dragging operations, providing a seamless user experience.

The system's implementation involves Python programming language, utilizing OpenCV for computer vision
tasks. Specifically, the MediaPipe package is employed for hand tracking, while Pynput, Autopy, and
PyAutoGUI packages facilitate window screen manipulation, including l right-click actions . Remarkably, the
proposed model achieves exceptional accuracy levels without necessitating a GPU, ensuring practicality in real-
world applications.

In essence, this innovative Virtual Mouse system marks a significant leap in HCI technology. By harnessing
the power of hand gestures and computer vision, it not only offers a futuristic interaction model but also aligns
with the global trend of minimizing physical interfaces. This system showcases the potential of intuitive, hands-
free computer interaction, paving the way for a future where traditional input devices become obsolete, and
users seamlessly control their digital world through gestures and cameras.

1.2 Motivation of Virtual Mouse:

It is fair to say that the Virtual Mouse will soon to be substituting the traditional physical mouse in the near
future, as people are aiming towards the lifestyle where that every technological devices can be controlled and
interacted remotely without using any peripheral devices such as the remote, keyboards, etc. it doesn't just
provides convenience, but it's cost effective as well.

1.2.1. Convenient :

It is known in order to interact with the computer system, users are required to use an actual physical
mouse, which also requires a certain area of surface to operate, not to mention that it suffers from cable
length limitations. Virtual Mouse requires none of it, as it only a webcam to allow image capturing of user's
hand position in order to determine the position of the pointers that the user want it to be.
For example, the user will be able to remotely control and interact the computer system by just facing the
webcam or any other image capturing devices and moving your fingers, thus eliminating the need to
manually move the physical mouse, while able to interact with the computer system from few feet away.

1|Page
 1.2.2. Cost Effective:

A quality physical mouse is normally cost from the range of 30 ringgit to a hefty 400 ringgit, depending on
their functionality and features. Since the Virtual Mouse requires only a webcam, a physical mouse are no
longer required, thus eliminating the need to purchase one, as a single webcam is sufficient enough to allow
users to interact with the computer system through it, while some other portable computer system such as the
laptop, are already supplied with a built-in webcam, could simply utilize the Virtual Mouse software without
having any concerns about purchasing any external peripheral devices.

1.3 Goals & Objective(s) of the work

The purpose of this project is to develop a Virtual Mouse application that targets a few aspects of significant
development. For starters, this project aims to eliminate the needs of having a physical mouse while able to
interact with the computer system through webcam by using various image processing techniques. Other than
that, this project aims to develop a Virtual Mouse application that can be operational on all kind of surfaces and
environment.

The following describes the overall objectives of this project:

• To design to operate with the help of a webcam. The Virtual Mouse application will be operational
with the help of a webcam, as the webcam are responsible to capture the images in real time. The
application would not work if there are no webcam detected.

• To design a virtual input that can operate on all surface. The Virtual Mouse application will be
operational on all surface and indoor environment, as long the users are facing the webcam while
doing the motion gesture.

• To program the camera to continuously capturing the images, which the images will be analysed, by
using various image processing techniques. As stated above, the Virtual Mouse application will be
continuously capturing the images in real time, where the images will be undergo a series of process,
this includes HSV conversion, Binary Image conversion, salt and pepper noise filtering, and more.

• To convert hand gesture/motion into mouse input that will be set to a particular screen position. The
Virtual Mouse application will be programmed to detect the position of the defined colours where it
will be set as the position of the mouse pointers.

Furthermore, a combination of different colours may result in triggering different types of mouse events, such
as the right/left clicks, scroll up/down, and more.

1.4 Implementation:

a. Camera
In virtual mouse, a camera serves as the primary input device by capturing continuous live video
frames.

b. OpenCv
OpenCV is a robust computer vision library, processes these frames in real-time through
techniques. These methods isolate specific objects, such as hands or markers, essential for gestures.
Algorithms within the virtual mouse software recognize these gestures, mapping them to precise mouse
actions like clicks or scrolls. This mapping, defined in the software, enables users to control the
computer's cursor seamlessly.

c. Pycharm
PyCharm, developed by JetBrains, stands as a favored integrated development environment
(IDE) for Python.
Its user-friendly interface and powerful tools facilitate Python development. The IDE boasts
intelligent code completion, robust debugging features, and syntax highlighting.

2|Page
 It supports renowned Python frameworks such as Django and Flask, making it especially
useful for web development projects. PyCharm integrates seamlessly with version control systems like
Git, streamlining collaboration efforts.
Additionally, it provides essential database tools and supports a variety of testing frameworks.
Catering to both beginners and seasoned developers, PyCharm is available in two editions: Community
(free) and Professional (paid), offering extensive features and flexibility for Python programmers.

OUTPUT

Before capturing gesture

Fig. 3 Before capturing gesture

After capturing gesture

Fig.4 Detecting of Palm/Hand

3|Page
Fig.5 Moving the Cursor

Fig.6 Double Click

4|Page
1.5 Algorithms and Methods
Algorithm:

Step1: Start

Step2: bring your finger iinfront of the Camera

Step3: Move your finger

Step4: For Double-click

Bring the two finger close to each other

Step5: repeat step3 and step 4

Step6: Exit

Data Flow Diagram:

Start

on

Camera

Detect
gesture

Move Cursor

Stop

Fig.1 . Data Flow diagram of Virtual Mouse

5|Page
 1.6 EXPERIMENTAL RESULT

After implementing the cursor on different task like Opening the file, closing the file, minimizing the file ,
maximizing the file and others , the accuracy of the cursor between 1-10 is rated in the below graph.

It has given a good a result while performing the task of opening the file , closing the file , minimising the
file , but while performing the task of maximizing of any file, its accuracy is not so good .

1.6.1 Problem Faced

While maximizing the file it is stuck in the loop of Minimizing and maximizing

Fig2. Graph of Accuracy of Cursor on performing different task

1.7 Future scope

The future of virtual mouse technology is brimming with potential. Ongoing advancements are set to
revolutionize human-computer interaction, with improvements in gesture recognition accuracy, 3D spatial
sensing, and customization options enhancing user experience.

Integration of multi-modal interactions like voice commands and eye-tracking promises more intuitive
interfaces. Ensuring paramount security, privacy, and compatibility across platforms is crucial.

Additionally, the technology's applications in gaming, healthcare, education, and industry are expanding,
transforming digital interactions. With the evolution of machine learning and AI, standard features may
include adaptive and predictive capabilities, refining the virtual mouse experience for users of diverse
backgrounds and abilities.

1.8 Conclusion:
The virtual mouse system presented in this paper signifies a significant leap in HCI technology. By
harnessing the power of computer vision and hand gestures, the system offers a truly device-free interaction
experience. As technology continues to advance, embracing innovative HCI solutions becomes crucial, ensuring
inclusivity and usability across diverse applications.

6|Page
 2. Literature Review
There are many related works which are carried out on Virtual mouse , few of them are given below.

[1] In 2010, Dung-Hua Liou, Chen Chiung Hsieh, and David Lee proposed a study titled 'A Real-Time Hand
Gesture Recognition System Utilizing Motion History Picture.' The main flaw in the proposed system lies in its
utilization of complex hand gestures."

[2]In their 2013 study titled 'Cursor Control System Using Hand Gesture Recognition,' Sneha U. Dudhane,
Monika B. Gandhi, and Ashwini M. Patil proposed an approach. The problem with this method is that the frame
must be saved before processing for detection, which takes significantly longer than real-time requirements.

[3] n 2014, Abhik Banerjee & Abhirup Ghosh proposed “Mouse Control using a Web Camera based on
Color Detection” [6] titled paper where the methodology is Hand gestures were acquired using a camera based
on color detection technique. The limitations of their work are the operating background has to be light and no
bright colored objects are present. It works well on certain computers of high configuration

[4] In 2016, Pooja Kumari, Saurabh Singh & Vinay Kr. Pasi developed “Cursor Control using Hand
Gestures” is based on multiple color bands where different colors perform different actions. The number of
colors is the key to perform mouse functions. But, multiple colors were used to control the system. It is relied on
the number of colors to perform a function instead of different gestures.

[5] In 2017, Aashni Haria, Archanasri Subramanian, Nivedhitha Asokkumar, Shristi Poddar, Jyothi S Nayak
developed “Hand Gesture Recognition for Human-Computer Interaction” based on background extraction and
contours detection system. But it is very slow to work with.

[6]In 2018, Naveen Wilson, Abhilash SS, Chaithanya C ,Lisho Thomas, and published a paper titled 'Virtual
Mouse Using Hand Gesture'. The system they designed operates based on color detection, with functions
dependent on the number of colors detected. However, it has limitations—it can perform only a limited set of
mouse.

7|Page
 3. Application
This project can be useful for presentations and for reducing workspace and burden of extra hardware devices.
In critical events such as battleground, operation theater, this project will be able to withstand itself, as it is
wispier than any other contemporary system in the market for computer interaction. Since it removes the
burdens of devices, it brings the user and the workspace more closer than before.

Robot controlling is one of the major applications of this system. Controlling robots without machines or
extra devices can be an excellent addition to this technology.

• Digital artist can draw 2D or 3D images using this gesture-based mouse on digital canvases. It will
allow more freedom more dynamic more space to artists to create their art.

• Critical events like a battleground, operation theater, mining fields can be controlled by gesture mouse.

• 0Virtual Reality or Augmented Reality based games can be played without any extended or wireless
devices with bare hands more comfortably.

• For patients with no limb control, this system can be proved very useful and effective.

• This mouse system can be applicable for sign language for the dumb & deaf people. It can help them to
interact with computing machines.

8|Page
 4. References
[1] C.-Chiung Hsieh, D.-Hua Liou, & D. Lee, , “A real time hand gesture recognition system using motion
history image,” Proc. IEEE Int'l Conf. Signal Processing Systems (ICSPS), 2. 10.1109/ICSPS.2010.5555462,
2010

[2]A. M. Patil1, S. U. Dudhane1, M. B. Gandhi1,” Cursor Control System Using Hand Gesture Recognition”,
International journal of advanced research in computer and communication engineering, vol. 2, issue: 5, May
2013

[3]Banerjee, A. Ghosh, K. Bharadwaj, “Mouse Control using a Web Camera based on Color
Detection,”IJCTT,vol.9, March 2014

[4]P. Kumari, S. Singh & V. Kr. Pasi, "Cursor Control using Hand Gestures," Recent Trends in Future
Prospective in Engineering & Management Technology 2016, International Journal of Computer Applications
(0975 – 8887), 2016

[5]A. Haria, A. Subramanian, N. Asokkumar, S. Poddar & J. S. Nayak, “Hand Gesture Recognition for
Human Computer Interaction," 7th International Conference on Advances in Computing & Communications,
ICACC-2017, Cochin, India, Procedia Computer Science 115 (2017), pp.367–374, 22-24 August 2017.

[6]S. S. Abhilash, L. Thomas, N. Wilson & C. Chaithanya, "VIRTUAL MOUSE USING HAND
GESTURE," International Research Journal of Engineering and Technology (IRJET), vol. 05 issue: 4, April
2018

[7] Chen-Chiung Hsieh, Dung-Hua Liou, David Lee “A real Time hand Gesture recognition system motion
history image” 2010 2nd International Conference on Signal Processing Systems (ICSPS)

[8] Sneha U. Dudhane, Monika B. Gandhi, and Ashwini M. Patil “Cursor Control System Using Hand
Gesture Recognition” International Journal of Advanced Research in Computer and Communication
Engineering Vol. 2 ,Issue 5.

[9] Naveen Wilson, Abhilash SS, Chaithanya C ,Lisho Thomas “'Virtual Mouse Using Hand Gesture”
International Research Journal of Engineering and Technology (IRJET), Volume: 05 Issue: 04 |.

[10] https://opencv.org/university/free-opencv-course

[11] https://en.wikipedia.org/wiki/PyCharm

9|Page