Gesture Palette : A Virtual Painting Website

T.Y.B.Tech.,Dept of CSE AIML KIT’s college of engineering (Autonomous) Kolhapur

Maharashtra,India

1st Sejal Mali 3rd Akanksha Arsekar

T.Y.B.Tech.,Dept of CSE AIML
KIT’s college of engineering
(Autonomous)
Kolhapur, Maharashtra, India
sejalmali2311@gmail.com
4th Siddhant Bodake
Dept of CSE AIML,
KIT’s college of engineering
(Autonomous)
Kolhapur, Maharashtra, India
line 5: siddhant07092003@gmail.com
7th Amit Chanchal
Asso.Prof., KIT’s college of
engineering (Autonomous)
Kolhapur, Maharashtra, India
chanchal.amitkumar@kitcoek.in
2nd Siddhi Yadav
T.Y.B.Tech.,Dept of CSE AIML
KIT’s college of engineering
(Autonomous)
Kolhapur, Maharashtra, India
siddhiyadav363@gmail.com,
5th Uma Gurav
Head, Department of CSE AIML DS
KIT’s college of engineering
(Autonomous)
Kolhapur, Maharashtra, India
gurav.uma@kitcoek.in
8th Vinay Prabhavalkar
Asst.Prof., KIT’s college of engineering
(Autonomous)
Kolhapur, Maharashtra, India
prabhavlkar.vinay@kitcoek.in
T.Y.B.Tech.,Dept of CSE AIML
, KIT’s college of engineering
(Autonomous)
Kolhapur, Maharashtra, India
akankshaarsekar1627@gmail.com,
6th Tanvi Patil
Asst.Prof., Dept of CSE AIML DS,
KIT’s college of engineering
(Autonomous)
Kolhapur, Maharashtra, India
patil.tanvi@kitcoek.in

Abstract— The era of digital technology has brought about
a new era where people with mobility challenges can access
digital content without touching a screen or any machine. In
this era, there is a growing demand for cleaner, more user-
friendly interfaces while extending accessibility to individuals
facing mobility challenges. The main aim of this research is to
create a revolutionary touchless digital canvas that offers a
fresh solution for hygiene, accessibility, and creative expression
needs. We delved into emerging concepts like haptic feedback
and 3D hand tracking, aiming to push the boundaries of user
interaction. We created a user interface that not only looks
good but also feels intuitive and engaging. We also
incorporated machine learning and AI techniques, reducing
errors and ensuring swift interpretations of user gestures. Our
project, rooted in deep learning and cutting-edge technologies,
not only builds upon prior research in human-computer
interaction but also offers a glimpse into a future where digital
interactions are not just technological but more inclusive and
creative, setting the high stage for a touchless Digital era.
Keywords— OpenCV, MediaPipe, Digital canvas, Gesture
recognition, Machine learning in touchless technology, Hygiene
in digital interactions.
I. INTRODUCTION
In today's fast-changing tech world, we use touch screens
and pens, but now things have changed, especially because
of worries about germs, like during the COVID-19
pandemic. As we navigate a post-pandemic era, the
imperative for touchless technology takes center stage. It
made us think about how we interact with our devices and
what we can do to make it cleaner and safer for everyone. In
this context, the Gesture Palette Website is not only a
response to current challenges but as a beacon illuminating
the path forward.
In our digital era, big changes are on the skyline nowadays
for how we interact with computers. We use tapping screens
and stylus, but these methods have some limitations and
issues that they’ve become less attractive due to concerns
about cleanliness and problems with accessibility as well as
hygiene. As the world changes, there’s a need for a new way
to interact with technology that doesn’t involve touching.
So, here comes our research Project introducing the
“Gesture Palette Website.” It’s an exciting solution that let
us draw and write without physically touching the screen.
But it’s more than just a technological advancement; it’s a
bridge to a future where our digital interactions are cleaner,
easier for everyone to use, and more creative. This research
paper doesn’t just talk about the development of this idea,
But it also explores where we could use it in various fields.
Think of it as going forward in how we deal with digital
stuff in the post-pandemic era. It’s a time when art and
technology work together seamlessly, and not touching
screens is the new usual way.
So, get ready for an adventure into a world where you can
do all the awesome stuff with your devices without worrying
about germs. The Gesture Palette Website is not just a cool
idea; it's like a story about making our digital world safer
and more fun, especially after the time when we all learned
the importance of keeping things clean.
II. RELATED WORK
The Authors of “Painting with a hand gesture using
Mediapipe” R. Vasavi and colleagues emphasize the
significance of hand gesture recognition, using the
MediaPipe framework and OpenCV to enable users to paint

and draw on a virtual canvas using fingertip movements.

 This system can recognize numbers from 0 to 9 devices are also used for this project like the DE2 board,
based on hand gestures. as well as this article also highlights camera, and VGA monitor, while the software is
the importance of computer vision and gesture recognition in implemented in Verilog. Meanwhile the research paper
enabling intuitive and natural interaction with the machines. ”Virtual Whiteboard-A Gesture Controlled Pen-free Tool”
Then the “Virtual paint application using hand gestures” discusses the development of an innovative application that
develops a virtual paint application that interprets hand combines computer vision and machine learning to create a
movements for drawing on screens and provides a virtual board controller using hand gestures. This paper also
comprehensive literature review of related work in hand uses the technique of fingertip outline detection, tracking,
gesture recognition, along with insights into the technologies and interaction with the virtual whiteboard. this paper
used. This application also introduces features for selecting highlights the use of image processing and pattern
and saving the work or drawings. it also includes the methods recognition techniques. Furthermore, it introduces
using marker detection, data gloves, and skin color detection. applications like PowerPoint controlling capabilities,
Move Further the “Virtual Painter using artificial intelligence allowing users to control presentations using hand gestures.
and OpenCV” introduces us to a real-time hand tracking The conclusion underlines the potential of virtual
system for drawing in front of a webcam, methods like hand whiteboards and hand gesture control as a revolutionary tool
region segmentation, background subtraction, hand centroid for education and communication
localization, and drawing lines using the position of the
outlines are used to produce the art through the AI.
Meanwhile, the “AI virtual painting on video calls using III. PROPOSED METHODOLOGY
CV” by The authors Alet Joseph Byju, Arjun K R, Delvin V
D, Mridhul M L, and guided by Ms. Neethu Prabhakaran
explores hand gesture recognition and motion capture,
focusing on real-time virtual drawing on video, aligning with
motion detection in video conferencing using image
sequences and deep learning techniques for gesture
recognition. Additionally, the project’s goal is real-time
virtual drawing on video call it also discusses about hand
tracking, motion detection from stereo image frames, and
WebRTC-based video conferencing systems with screen
sharing capabilities. These related works provide valuable
insights into the technological components required for AI-
powered hand gesture recognition and motion capture in
video conferencing settings, which is the central theme of
this research paper.
Furthermore “Virtual AI painter using OpenCV and
media pipe” authored by Akshat Rastogi, Aryan Aggarwal,
Ayushi Vashistha, and Anmol Rastogi from the Department
of Computer Science and Engineering at Meerut Institute Of
Engineering And Technology presents a creative platform for
real-time virtual drawing, it provides the applications in
communication, digital art, and education so it could be A. Conceptualization
accessible and handle by the every age person. Finally, The project’s journey begins with the exciting phase of
“Virtual sketch using openCV” the study conducted by conceptualization, where the core idea of the Gesture Palette
Pranavi Srungavarapu, Eswar Pavan Maganti, Srilekkha Website takes place. In this phase, we dive into the world of
Sakhamuri, Sai Pavan Kalyan Veerada, and Anuradha virtual interaction to discover its full potential.
Chinta. The authors present an innovative project that 1) Let’s explore how we laid the groundwork:Touch
enables interactive sketching using a colored marker and
less Interaction Concepts: We step board on an
computer vision. It offers an interactive sketching tool using
Python and OpenCV, a project that enables interactive exploratory journey into the interesting realm of touch less
sketching using a colored marker and computer vision interaction. Beyond the well-known technologies like
including Computer Science and Engineering. Mohd gesture recognition, we delved into emerging concepts
Sallehudin bin Saad initiated a thesis ”Virtual paint on de2 like haptic feedback and 3D hand tracking, aiming to push
FPGA board through gesture detection and color the boundaries of user interaction.
segmentation technique” This research project focuses on the 2) Design Possibilities: Our design process was not
development of a Virtual Paint system, inspired by the ”Sixth just about aesthetics but also about functionality and
Sense” technology. This system allows users to draw colored usability. We brainstormed and explored design
images on flat surfaces using hand movements and fingers possibilities that went beyond the conventional, aiming to
equipped with colored bands, eliminating input devices like a create a user interface that not only looks good but also
mouse or joystick. The project is developed by using the
feels intuitive and engaging.
integration of gesture detection and color segmentation
techniques. The hardware implementation is based on the 3) Functional Scope: It’s crucial to know what our
Altera DE2 FPGA board, with Verilog code used for project can do. In defining the functional scope, we
programming. The system utilizes a 5-megapixel camera and aimed to create a clear roadmap for the Gesture Palette
a VGA monitor to capture hand movements and give access Website.We outlined features such as collaborative
to the user for drawing in the air. for extracting gesture drawing, virtual whiteboard capabilities, and interactive
detection and color segmentation they used a CMOS camera
sensor and implemented it using Verilog. Some hardware
 touch less tools, ensuring that our project would be as
versatile as possible.
B. Choice of Web Framework
Selecting the web framework was a crucial moment. We
needed a robust, flexible, and scalable framework to turn
our vision into reality. After careful consideration, we
choose Flask, the Python web framework. Its versatility
and strong community support provided the perfect
canvas for our project.
C. Technological Stack
a) Let’s take a closer look at the components:HTML
and CSS: In the world of web development, HTML and
CSS are like the artist’s palette and canvas. They allow
us to craft beautiful, responsive,and interactive user
interfaces.
b)JavaScript: Think of JavaScript as the magician’s
stick, bringing interactivity to our website. It makes
real-time updates possible,ensuring that our users have
a dynamic and engaging experience.
c) Python: Our chosen back-end language, Python, is
like the conductor of an orchestra. Through Flask,
Python adapts the server-side logic,ensuring that
everything runs smoothly and efficiently.
blend of technologies enabled our model to decipher
gestures with exceptional accuracy.
3) Training and Testing: Our model goes through
extensive training and testing, fine-tuning its
capabilities. The result? A responsive touchless
interaction experience that empowers users to create
effortlessly.
G. Touchless Interaction and Hand Gestures
The touchless interaction module is where the magic
happens. Imagine your fingertip as a brush, and the
canvas as a digital masterpiece on the website.
Techniques like canvas manipulation and event handling
allow users to create, write, and draw with the help of
their gestures. Through innovative techniques such as
canvas manipulation and event handling, users can
weave their creativity, crafting a digital masterpiece with
the sheer magic of their gestures.

D. Authentication
In the digital world, security and user management has
supreme importance. We incorporated robust
authentication to create a safe and collaborative
environment for our users. Think of it as the keys to a
private studio. Users can access their tools securely and
collaborate effectively. IV. RESULT ANALYSIS
 HERE ARE THE FINEL RESULT OF OUR WORKING PROJECT
E. Database Integration and User Management
The MySQL database system was selected as our
information vault. It’s like a digital sketchbook, storing
user information securely and efficiently. Through it, we
maintain user profiles, drawings, and interactions,
ensuring a seamless and personalized experience for
each user

F. Integration of Machine Learning

To promote touchless interactions, we integrated a
machine-learning model. Picture this model as the
project’s wizard, learning from the user’s movements Here is the first picture of our working project. By showing
and gestures. Through deep learning techniques and this sign model can detect the hand of human and activate
neural networks, it accurately recognizes the gestures and the function of writing.
predicts user intent

1) Data Collection: Just as artists gather inspiration

from the world around them, our model collected
vast datasets of touchless interactions. These
datasets included an array of gestures, from simple
hand movements to complex patterns.
2) Model Development: Our deep learning model was
crafted with precision, incorporating convolutional
and recurrent neural networks. This sophisticated
This is the second gesture with the help of this one finger we
can write or draw anything on the screen. By only moving
our finger in the way, as usual, we can write. As we go on the next page, we also go back to the previous
page by showing this sign.

By using this sign model can able to know that we are We can draw anything on a screen virtually with the help
selecting or checking our required size, color, or eraser. This of only our fingertip which is shown in this picture
sign is used to confirm our selection

This is the sign from which we can resize our curser or

pointer like thin or thick and then we use the check sign to As shown in this picture our project is fully working and
confirm our selection anyone can write anything on the screen easily.

V. FLOWCHART
This flowchart explains the procedure or working of our
website step by step.

When we want to use the next page that time simply use this
sign. this is help to go to the next page.
 1.One Finger UP
This hand gesture is used to draw
anything on the screen

2.Two Fingers UP
This hand gesture is used to either
select colors from color palette or
change the position of hand on
the screen

 When the User logs in or sighed up to the website it will 3.ThumbAnd IndexFinger UP
grant access to the webcam then the actual work of our This hand gesture is used to
project will begin increase /decrease the size of the
 Then the model detects the hand gesture of the user stroke
whether it is one finger up, Two fingers up, or a Thumb
pointing towards left or right
 According to the gestures it will draw the user-required
output on the screen 4.Thumb,Index and
 After completion of work user will able to save the Pinky finger up
work This Hand Gesture is Used To
 And log out the webpage check expanded stroke

VI. OUR CONTRIBUTION

5.Thumb Directing Towards
 As we see lots of paper at that time, we realize that right
there is a lack of colors to visualize the amazing user This hand Gesture is used to
experience. So we have added some more color to over switch to the next page
"Gesture Palette” website.
Also, we have provided the save option where we can
save overwork in our local system by just moving
overhand to the save button. You will be more clear by
seeing this picture. 6.Thumb Directing Toward
Left
This hand gesture is used to
switch to the previous page

VII. FUTURE PLANS

 This is the picture of the top side of the website here are
3 options
1. We can see the page number we are working on.
2. We have easily accessible Color Pallete and eraser
also to erase the not required work.
3. Also we have added the save button at the top right
corner beside the logout button with the help of that
we can save our work in the PNG format.
In our project, we have added new gestures to the
model which help the model be easily trained on
what exactly the user wants to draw or write on the
screen.
Our focus lies in advancing practical experiences across
various domains. We will give new and improved forms,
and structure, overall, we aim to create a more engaging
platform. Then we are also planning to add Augmented
Reality (AR) features by Switching gears to cross-platform
communication like mobile devices, also We will develop
apps for both iOS and Android. We are trying to do Picture
virtual classrooms; it will give an interactive learning
platform to the students with additional features like
Multiple users interacting with a canvas. simultaneously it
also helps to collaborate and enhance virtual meetings. And
let's not forget about entertainment and recreation. Crafting
experiences in gaming, where players control characters and
 environments with gestures. We are going to offer them a
new and exciting dimension to play. Lastly, the future is a
continuous journey of technical improvement but
Collecting feedback from users regularly, understanding
their needs and wants, and using this information to
iterate technology, resolve issues, and add new features is
important so we decided to give the option of feedback.
VIII. CONCLUSION
In an ever-evolving digital landscape, the “Gesture Palette
Website” emerges as a pioneering solution to redefine how
we interact with technology. In a world increasingly worried
about hygiene and accessibility, this touchless digital canvas
presents a transformative approach. By enabling users to
draw and write without being in contact with any digital
accessories. It addresses the demand for cleaner, more user-
friendly interfaces while extending accessibility to
individuals facing mobility challenges. This project, rooted
in deep learning and cutting-edge technologies, not only
builds upon prior research in human-computer interaction
but also offers a glimpse into a future where digital
interactions are not just technologically but more inclusive
and creative, setting the high stage for a touchless digital
era.
IX. REFERENCES
[1] Saad, M.S., 2014. Virtual Paint on DE2 FPGA Board Through
Gesture Detection and Color Segmentation Technique (Doctoral
dissertation, Universiti Tun Hussein Onn Malaysia).
[2] Vasavi, R., Rahul, N., Snigdha, A., Moses, K.J. and Simha, S.V.,
Painting with Hand Gestures using MediaPipe.
[3] Niharika M, Neha J, Mamatha Rao, Vidyashree K P., VIRTUAL
PAINT APPLICATION USING HAND GESTURESY.
[4] Rastogi, A., Aggarwal, A., Vashistha, A. and Rastogi, A., VIRTUAL
AI PAINTER USING OPENCV AND MEDIAPIPE
[5] Bansode, S., Varkhad, S., Dhaigude, S., Waghmare, S. and
Suryawanshi, S., Computer Vision Based Virtual Sketch Using
Detection.