CONTROL WITH HAND GESTURES

Submitted by

S.AJITH KUMAR(Reg. No.: 1612004)

M.AJOY DONISON(Reg. No.: 1612005)

P.Karthick Raja(Reg. No.: 1612046)

 ABSTRACT

Hand gesture is an innovative technique of interaction between human and computer. As compared to
existing techniques, hand gesture technique has the advantage of being easy to use. By using this technique the
traditional way of using mouse and keyboard will be change because one can then interact with the computer
with hand gestures. In this technique, ultrasonic sensor is used to classify the hand movement in real-time.
Ultrasonic sensor measures the distance of hand by using sound waves. The main idea of our approach is to
speed up the interaction with computer, using general purpose hardware like personal computer, Arduino UNO
board and low cost sensor like ultrasonic sensor. In this way any user can easily interact with the computer
using hand gestures.

i
 TABLE OF CONTENTS

1 INTRODUCTION… .............................................................................................. 1

2 PROPOSED SYSTEM… ......................................................................................... 2

2.1 COMPONENTS REQUIRED ..................................................................... 2

2.2 CIRCUIT DIAGRAM ................................................................................. 3

2.3 MODULE DESCRIPTION ......................................................................... 3

2.3.1 ARDUINO UNO .............................................................................. 3

2.3.2 ULTRASONIC SENSOR ................................................................ 4

2.4 WORKING ................................................................................................. 5

3 IMPLEMENTATION ........................................................................................... 7

3.1 PROGRAMMING THE ARDUINO TO DETECT GESTURES ............. 7

3.1.1 ARDUINO CODE ........................................................................ 7

3.2 PYTHON PROGRAMMING FOR THE PROJECT ................................. 11

3.2.1 PYTHON CODE .......................................................................... 11

3.3 SNAPSHOTS............................................................................................ 13

4 CONCLUSION .................................................................................................... 14

ii
 CHAPTER-1

INTRODUCTION

Today’s world is a high technological world; imagining life without computer is

impossible. Now computer is an essential part of human’s day-to-day life. Vision and gesture are the important
approach for communication between human beings just in the same way as keyboard and mouse play a role for
interacting with computer. Numbers of effective techniques are available for interaction with computer and one
of them is hand gesture technique. In that technique hand gesture is used as an input which replaces
functionality of mouse and keyboard shortcut keys. Hand gesture is an attractive and faster technique. For
determining user hand distance ultrasonic sensors are used. By using hand gesture user can communicate with
computer easily and there is no need of any physical connection between user and system.

The concept behind the project is very simple. We will place two Ultrasonic (US) sensors
on top of our monitor and will read the distance between the monitor and our hand using Arduino, based on this
value of distance we will perform certain actions. To perform actions on our computer we use Python pyautogui
library. The commands from Arduino are sent to the computer through serial port (USB). This data will be then
read by python which is running on the computer and based on the read data an action will be performed.

Human Machine Interface or HMI is a system comprising of hardware and software that helps in
communication and exchange of information between the user (human operator) and the machine. Instead of
using a keyboard, mouse or joystick, we can use our hand gestures to control certain functions of a computer
like play/pause a video, move left/right in a photo slide show, scroll up/down in a web page and many more. In
this project, we have implemented a simple Arduino based hand gesture control where you can control few
functions of increase or decrease the volume (in video Player),seek forward and backward, scroll through pages
in a pdf with the help of hand gestures.

1
 CHAPTER-2

PROPOSED SYSTEM

This paper introduces a technique based on determining distance by the sensor and accordingly a
particular function is performed. Some recognition method of the gestures are proposed and then actions are
recognized using sensor .We set up few mainstream methods based on the action recognition by the sensors.
The sensor device is attached on computer at head of the screen, for quick operation. In this field much research
work has been done but that work is related to hand recognition, real time finger recognition and recognition of
alphabet characters. Real time human computer interaction using hand gesture, are also used for many
functionalities such as video control, music player, gaming, controlling the functions of PDF reader etc. All
these interactions have real time gesture recognition techniques. A gesture controller resolution always requires
a physical device which follows and recognizes the body language or movements, so that the computer can
clarify them. By using ultrasonic sensor, the distance of hand can be found which acts as an input. According to
the distance of hand, particular function is performed.

2.1 COMPONENTS REQUIRED

HARDWARE REQUIREMENTS

• Arduino UNO
• Ultrasonic Sensors x 2
• USB Cable (for Arduino)
• Few Connecting wires
• A Laptop with internet connection

SOFTWARE NEEDED

• ARDUINO IDE
• Python (pyautogui)

2
2.2 CIRCUIT DIAGRAM

Fig 2.2.1

2.3 MODULE DESCRIPTION

2.3.1 ARDUINO UNO

Fig 2.3.1.1

3
The Arduino UNO is an open-source microcontroller board based on the Microchip ATmega328P
microcontroller and developed by Arduino.cc. The board is equipped with sets of digital and analog
input/output (I/O) pins that may be interfaced to various expansion boards (shields) and other circuits. The
board has 14 Digital pins, 6 Analog pins, and programmable with the Arduino IDE (Integrated Development
Environment) via a type B USB cable. It can be powered by a USB cable or by an external 9 volt battery,
though it accepts voltages between 7 and 20 volts. It is also similar to the Arduino Nano and Leonardo. "Uno"
means one in Italian and was chosen to mark the release of Arduino Software (IDE) 1.0. The Uno board and
version 1.0 of Arduino Software (IDE) were the reference versions of Arduino, now evolved to newer
releases.The Uno board is the first in a series of USB Arduino boards, and the reference model for the Arduino
platform. The ATmega328 on the Arduino Uno comes preprogrammed with a bootloader that allows uploading
new code to it without the use of an external hardware programmer.It communicates using the original STK500
protocol. The Uno also differs from all preceding boards in that it does not use the FTDI USB-to-serial driver
chip. Instead, it uses the Atmega16U2 (Atmega8U2 up to version R2) programmed as a USB-to-serial
converter.

2.3.2 ULTRASONIC SENSOR

Fig 2.3.2.1

4
Ultrasonic transducers or ultrasonic sensors are a type of acoustic sensor divided into three broad categories:
transmitters, receivers and transceivers. Transmitters convert electrical signals into ultrasound, receivers convert
ultrasound into electrical signals, and transceivers can both transmit and receive ultrasound.

In a similar way to radar and sonar, ultrasonic transducers are used in systems which evaluate targets by
interpreting the reflected signals. For example, by measuring the time between sending a signal and receiving an
echo the distance of an object can be calculated. Passive ultrasonic sensors are basically microphones that detect
ultrasonic noise that is present under certain conditions.

2.4 WORKING
Gesture controlling is based on specifying hand position from the ultrasonic sensor. For
processing the raw data, a micro-controller is essential; for that we use Arduino UNO board. Via USB
connection the microcontroller transfers the processed and calculated distance value which is provided by the
sensor. The data which is send by the sensor is processed in the software in PC where all the calculations are
performed and the data is matched with the predefined conditions (gesture resolution). In this model two
ultrasonic sensors are used to detect hand position and are connected to the Arduino board .As we know
ultrasonic sensor continuously emits sound and it gets reflected back from user’s hand. The distance between
the sounds is send and detection of reflect back sound wave is calculated by the micro-controller.

Fig 2.4.1

5
 Fig 2.4.2
This model has following hardware components; these are two ultrasonic sensor (HCSR-04) and Arduino UNO
board. In figure 4 shows the circuit diagram of model. System software includes Arduino IDE and python GUI.
To run this model, the python code should run on python GUI first.It matchesgestures with predefined
conditions and prints on python output shell as shown in figure:
i). Interaction between python and Arduino program happens by writing arduino port number into
python program. Table No.1 shows the mapping between the hand gestures and output produced by the system.

6
 CHAPTER 3

IMPLEMENTATION
3.1 PROGRAMMING THE ARDUINO TO DETECT GESTURES

The important part of this project is to write a program for Arduino such that it converts the distances measured
by both the sensors into the appropriate commands for controlling certain actions. The hand gestures in front of
the Ultrasonic sensors can be calibrated so that they can perform five different tasks on your computer.

Hand Gesture Output Produced by system

Video PDF
Right hand push in Forward Page up
Right hand pull out Volume up Home
Left hand push in Rewind Page Down
Left hand pull out Volume down End

3.1.1 ARDUINO CODE

const int trigPin1 = 2;
const int echoPin1 = 3;
const int trigPin2 = 5;
const int echoPin2 = 4;
long duration;
int distance1, distance2;
float r;
unsigned long temp=0;
int temp1=0;
int l=0;

void find_distance (void);

7
void find_distance (void)
{
digitalWrite(trigPin1, LOW);
delayMicroseconds(2);
digitalWrite(trigPin1, HIGH);
delayMicroseconds(10);
digitalWrite(trigPin1, LOW);

duration = pulseIn(echoPin1, HIGH, 5000);

r = 3.4 * duration / 2;
distance1 = r / 100.00;

digitalWrite(trigPin2, LOW);
delayMicroseconds(2);
digitalWrite(trigPin2, HIGH);
delayMicroseconds(10);
digitalWrite(trigPin2, LOW);

duration = pulseIn(echoPin2, HIGH, 5000);

r = 3.4 * duration / 2;
distance2 = r / 100.00;
delay(100);
}

void setup()
{
Serial.begin(9600);
pinMode(trigPin1, OUTPUT);
pinMode(echoPin1, INPUT);
pinMode(trigPin2, OUTPUT);
pinMode(echoPin2, INPUT);
8
 delay (1000);

void loop()
{
find_distance();

if(distance2<=35 && distance2>=15)

{
temp=millis;
while(millis()<=(temp+300))
find_distance();
if(distance2<=35 && distance2>=15)
{
temp=distance2;
while(distance2<=50 || distance2==0)
{
find_distance();
if((temp+6)<distance2)
{
Serial.println("down");
}
else if((temp-6)>distance2)
{
Serial.println("up");
}
}
}
else
{
Serial.println("next");
9
 }
}

else if(distance1<=35 && distance1>=15)

{

temp=millis();

while(millis()<=(temp+300))
{
find_distance();
if(distance2<=35 && distance2>=15)
{
Serial.println("change");
l=1;
break;
}
}

if(l==0)
{
Serial.println("previous");
while(distance1<=35 && distance1>=15)
find_distance();
}
l=0;
}

10
3.2 PYTHON PROGRAMMING FOR THE PROJECT

Writing Python Program for Arduino based Hand Gesture Control is very simple. You just need to read the
Serial data from Arduino and invoke certain keyboard key presses. In order to achieve this, you have to install a
special Python Module called PyAutoGUI. With the help of PyAutoGUI, we can write a Python Program to
mimic the actions of mouse like left click, right click, scroll, etc. and keyboard like keypress, enter text,
multiple key press, etc. without physically doing them. The Arduino sends out five different texts or commands
through Serial Port upon detecting appropriate hand gestures. These commands are

• Next
• Previous
• Down
• Up

Using these commands along with few functions in PyAutoGUI (like hotkey, scroll, keyDown, press and
keyUp), you can write a simple Python Code that will execute the following tasks of keyboard and mouse.

• Data = “next” – – > Action = Ctrl+PgDn

• Data = “previous” – – > Action = Ctrl+PgUp
• Data = “down” – – > Action = Down Arrow
• Data = “up” – – > Action = Up Arrow

3.2.1 PYTHON CODE

import serial
import time
import pyautogui
print("Enter the Choice")
print("1.PDF mode 2.Video Mode")
ch=int(input())
Arduino_Serial = serial.Serial('com3',9600)
time.sleep(2)
if(ch==1):
while 1:
11
 incoming = str (Arduino_Serial.readline())
print (incoming)
if 'vup' in incoming:
pyautogui.hotkey('down')
if 'previous' in incoming:
pyautogui.hotkey('pgup')
if 'next' in incoming:
pyautogui.hotkey('pgdn')
if 'down' in incoming:
pyautogui.hotkey('end')
if 'up' in incoming:
pyautogui.hotkey('home')
incoming = "";
elif(ch==2):
while 1:
incoming = str (Arduino_Serial.readline())
print (incoming)
if 'vup' in incoming:
pyautogui.typewrite(['space'], 0.2)

if 'previous' in incoming:
pyautogui.hotkey('ctrl', 'left')

if 'next' in incoming:
pyautogui.hotkey('ctrl', 'right')
if 'down' in incoming:
pyautogui.hotkey('ctrl', 'down')
if 'up' in incoming:
pyautogui.hotkey('ctrl', 'up')
incoming = "";
else:
print("Invalid")
12
3.3 SNAPSHOTS

Fig 3.3.1

Fig 3.3.2

13
 CHAPTER 4

CONCLUSION

It is one of the easiest way of interaction between human and computer .It is a cost effective model
which is only based on Arduino UNO and ultrasonic sensor. The python IDE allows a seamless integration with
Arduino UNO in order to achieve different processing and controlling methods for creating new gesture control
solutions. The gesture recognition portion of our algorithm is too simple, and would need to be improved if this
technique would need to be used in challenging operating conditions. Reliable performance of hand gesture
recognition techniques in a general setting require dealing with occlusions, temporal tracking for recognizing
dynamic gestures, are still mostly beyond the current state of the art.The use of real embedded OS could
improve our system in terms of speed and stability. In addition, implementing more sensor modalities would
improve robustness even in very complex scenes

14
 REFERENCE
[1] Cost effective hand gesture computer control interface ,GergelySziládi; Tibor Ujbányi; JózsefKatona2016
7th IEEE International Conference on Cognitive Infocommunications
[2] Vision based computer mouse control using hand gestures,SandeepThakur; Rajesh Mehra; Buddhi Prakash,
2015 International Conference on Soft Computing Techniques and Implementations (ICSCTI)
[3] A basic hand gesture control system for PC applicationsC. J. Cohen; G. Beach; G.FoulkProceedings 30th
Applied Imagery Pattern Recognition Workshop (AIPR 2001). Analysis and Understanding of Time Varying
Imagery
[4] A system for controlling personal computers by hand gestures using a wireless sensor vice, Kaoru magishi;
Lei Jing; Zixue Cheng2014 IEEE International Symposium on Independent Computing (ISIC)
[5] CMSWVHG control MS Windows via hand gesture, Hafız Mohsin Abdul-Rashid; Lehmia Kiran; M.
Danish Mirrani; M. Noman Maraaj2017 International Multi-topic Conference (INMIC)
[6] Power point control using hand gesture recognition based on hog feature extraction and k-nn classification
,Tejashree P. Salunke; S. D. Bharkad , 2017 International Conference on Computing Methodologies and
Communication (ICCMC)

15