A

Seminar Report
on
GESTURE BASED ROBOTIC VEHICLE USING SIXTH
SENSE TECHNOLOGY

By
Vasu Jain (1609131176)

DEPARTMENT OF ELECTRONICS
AND COMMUNICATION
ENGINEERING
JSS ACADEMY OF TECHNICAL EDUCATION
C-20/1 SECTOR-62, NOIDA
[2019-2020]

1
 A
Seminar Report on
GESTURE BASED
ROBOTIC VEHICLE
USING SIXTH SENSE
TECHNOLOGY

In partial fulfillment of requirements for the degree of

Bachelor of
Technology in
Electronics and
Communication
Engineering
SUBMITTED BY:
VASU JAIN
Under the Guidance of
Surekha Bhangari

DEPARTMENT OF ELECTRONICS
AND COMMUNICATION
ENGINEERING
JSS ACADEMY OF TECHNICAL EDUCATION
C-20/1 SECTOR-62, NOIDA
[2019-2020]

2
 CERTIFICATE

Certified that seminar work entitled “Gesture based robotic vehicle using sixth sense
technology ” is a bonafide work carried out in the eight semester by Vasu Jain in partial
fulfilment for the award of Bachelor of Technology in Electronics and Communication
Engineering from JSS Academy of Technical Education, Noida during the academic year
2019- 2020.

SIGNATURE

Surekha Bhangari

3
 TABLE OF CONTENTS PAGE NO.

ABSTRACT 5

1. INTRODUCTION 6
2. ORIGIN OF IDEA 7
3. OVERVIEW 8
4. DETAILED DESCRIPTION 9
5. FUNCTIONAL FLOWCHART 10
6. IMPLEMENTATION 16
7. SIMULATION RESULT 18
8. APPLICATION 21
9. CONCLUSION 23
BIBLIOGRAPHY 24

4
 ABSTRACT

Sixth Sense technology is a technology with which a system could be trained to recognize and
percept real world objects and react as desired. Sixth Sense technology bridges the gap
between the physical world and the digital world, bringing intangible, digital information out
into the tangible world, and allowing us to interact with this information via natural hand
gestures. Sixth Sense Technology is implemented in 'Sixth Sense/WUW (wear your world)
using gesture recognition, augmented reality, computer vision and radio frequency
identification. It’s just born concept which allows user to connect with the internet
seamlessly. Without use of keyboard, mouse we can see videos access, change, move data
simply. But this concept bottle necks lead to modification of the same by using commands
instead of gestures. Sixth Sense technology could be integrated with voice recognition.
Bluetooth device and laser projectors could be used.
The way of interaction between human and robot has been developing using various
technologies. Controlling the robotic vehicle using gestures is a predominant way of
enhancing the interaction. In this interactive technique the user need not have any physical
contact with the device. It helps to bridge the technological gap in the interactive system. In
this project the robotic vehicle is directed by identifying the real time gesture commands
from the user which is implemented using image processing algorithms and embedded
techniques.

5
 1. INTRODUCTION

Human gesture enhances human-robot interaction by making it independent

from input devices. Using gestures provides a more natural way of controlling
and provides a rich and intuitive form of interaction with the robot. The main
purpose of gesture recognition research is to identify a human gesture and
convey information to the user pertaining to individual gesture. From the
corpus of gestures, specific gesture of interest can be identified and based on
that, specific command for execution of action can be given to robotic system.

A prominent benefit is that it presents a natural way to send information to the

robot such as forward, backward, left and right movements etc. In order to
ease a feasible solution for user friendly interface, user can give commands to
a wireless robot using hand gestures. The early device was mainly for
navigation and controlling robot without any natural medium. This paper deals
with interface of robots through gesture-controlled technique but far away
from the user. This can be achieved through image processing technique.
Hand Gesture Recognition technology is implemented using „Data Gloves‟
which in turn leads to additional cost and lack of availability among majority
of masses.

Webcam is an easily available device and today every laptop has an integrated
webcam along with it. In this project, hand gesture recognizer which can
detect a moving hand with its gesture in webcam frames is implemented. In
future, it may be considered that willing to be more natural and more
comforted. Command is generated at the control station and sent to the robot
through ZigBee within the ZigBee range. The robot moves in the specified
direction according to the specified command.

6
 2. ORIGIN OF IDEA

WUW was developed by Pranav Mistry, a Ph. D student at Fluid Interfaces

Group at the MIT Media Lab. The Sixth Sense prototype implements several
applications that demonstrate the usefulness, viability and flexibility of the
system acts as the computer and your connection to the Cloud, all the
information stored on the web. Sixth Sense recognizes the objects around you,
displaying information automatically and letting you access it in any way you
want, in the simplest way possible. The device brings us closer to reality and
assists us in making right decisions by providing the relevant information,
thereby, making the entire world a computer. The technology is mainly based
on hand gesture recognition, image capturing, processing, and manipulation,
etc. The software of the technology uses the video stream, which is captured
by the camera, and also tracks the location of the tips of the fingers to
recognize the gestures. This process is done using some techniques of
computer vision. He invented ‘Sixth Sense / WUW (Wear UR World)’ which
is a wearable gestural , user friendly interface which links the physical world
around us with digital information and uses hand gestures to interact with
them. This technology is a revolutionary way to interface the physical world
with digital information. Modern technologies include the touch screen
techniques which is used widely, and it makes ease of operation and saves
utilization time.

7
 3. OVERVIEW

Previously many technologies evolved such as augmented reality which is to

add information and meaning to real object or place. Unlike virtual reality,
augmented reality does not create a simulation of reality instead it takes a real
object or space as the foundation and incorporates technologies that add
contextual data to deepen a person understanding of the subject. It’s a term for
live direct or indirect view of a physical real-world environment whose
elements are augmented by virtual computer-generated imagery. Gesture
recognition is a term with a goal of interpreting human gestures through
mathematical gestures and mathematical algorithms. Computer vision is the
science and technology of machines that is concerned with the theory behind
artificial systems that extract information from the images. As a technological
discipline, computer vision seeks to apply its theories and models to the
construction of computer vision systems. The examples include the controlling
processes, detecting events, organising information, modelling objects or
environments and interaction. Recently speech integrated circuits evolved
which is used widely in car automation and home appliances. It eases the
operation and saves the utilization time of the manual operations performed by
the human’s every day. The speech recognition process is performed by a
software component known as speech recognition engine. The primary
function of this is to process the spoken input and translate it into text which
the application understands. The application can interpret the result of the
recognition as a command, in this case application is a command and control
application. If the application handles the recognized text as simply text, then
it’s considered as dictation application. When the user says something, it is
known as utterance. An utterance is a stream of speech between two periods of
silence. The speech IC can be used for all sorts of data, statistical models, and
algorithms to convert spoken input into text.

8
 4. DETAILED DESCRIPTION

4.1 COMPONENTS:

The components are:

1. Camera

2. Projector

3. Mirror

4. Mobile component

5. Colour markers

6. Microphone

4.1.1Camera

Camera captures an object in view and tracks the user’s hand gestures. It sends the data to
smart phone. Camera recognizes and tracks user's hand gestures and physical objects using
computer-vision based techniques. Sixth Sense system implements a gestural camera that
takes photos of the scene the user is looking at by detecting the ‘framing’ gesture. It acts as a
digital eye, connecting you to the world of digital information.

Figure 1 :Camera

9
4.1.2 Projector

A tiny LED projector displays data sent from the smart phone on any surface in view-object,
wall, or person. The projector projects visual information enabling surfaces, walls and
physical objects around us to be used as interfaces. The projector projects visual information
enabling surfaces, walls and physical objects around us to be used as interfaces. We want this
thing to merge with the physical world in a real physical sense. You are touching that object
and projecting info onto that object. The information will look like it is part of the object. The
projector itself contains a battery inside, with 3 hours of battery life.

Figure 2 :Projector

4.1.3 Mirror

The mirror reflects the projection coming out from the projector and thus helps in projecting
onto the desired locations on walls or surfaces. The user manually can change the tilt of the
mirror to change the location of the projection. For example, in application where the user
wants the projection to go on the ground instead of the surface in front, he can change the tilt
of the mirror to change the projection. Thus, the mirror in the Sixth Sense helps in
overcoming the limitation of the limited projection space of the projector.

10
 Figure3: Mirror

4.1.4 Mobile component

The Sixth Sense system uses a mobile computing device in user’s pocket as the processing
device. The software program enabling all the features of the system runs on this computing
device. This device can be a mobile phone or a small laptop computer. The camera, the
projector and the microphone are connected to this device using wired or wireless connection.
The detail of the software program that runs on this device is provided in next section. The
mobile computing device is also connected to the Internet via 3G network or wireless
connection. A Web-enabled smart phone in the user's pocket processes the video data, using
vision algorithms to identify the object. Other software searches the Web and interprets the
hand gestures.

11
 Figure 4: Smartphone

4.1.5 Colour marker

Colour marker is at the tip of the user’s fingers. Marking the user’s fingers with red, yellow,
green, and blue tape helps the webcam recognize gestures. The camera tracks the movements
of the colour markers. The movements and arrangements of these makers are interpreted into
gestures that act as interaction instructions for the projected application interfaces. The
software program processes the video stream data captured by the camera and tracks the
locations of the coloured markers (visual tracking fiducials) at the tip of the user’s fingers
using simple computer-vision techniques. The movements and arrangements of these
fiducials are interpreted into gestures that act as interaction instructions for the projected
application interfaces. The maximum number of tracked fingers is only constrained by the
number of unique fiducials, thus Sixth Sense also supports multi-touch and multi-user
interaction.

Figure 5: Colour marker

12
4.1.6 Microphone

The microphone is an optional component of the Sixth Sense. It is required when using a
paper as a computing interface. When the user wants to use a sheet of paper as an interactive
surface, he or she clips the microphone to the paper. The microphone attached this way
captures the sound signals of user’s touching the paper. This data is passed to computing
device for processing. Later, combined with the tracking information about user’s finger, the
system is able to identify precise touch events on the paper. Here, the sound signal captured
by the microphone provides time information whereas the camera performs tracking. The
applications enabled by this technique are explained earlier.

Figure 6: Microphone

4.2 WORKING OF SIXTH SENSE TECHNOLOGY

The hardware that makes Sixth Sense work is a pendant like mobile wearable interface. It has
a camera, a mirror and a projector and is connected wirelessly to a Bluetooth smart phone
that can slip comfortably into one’s pocket. The camera recognizes individuals, images,
pictures, gestures one makes with their hands. Information is sent to the Smartphone for
processing. The downward-facing projector projects the output image on to the mirror. Mirror

13
reflects image on to the desired surface. Thus, digital information is freed from its confines
and placed in the physical world.

Figure7:

14
 5. FUNCTIONAL FLOWCHART
Inbuilt webcam of the system is used to capture the real time video of gestures. Colour
markers are used to interpret the gesture movements. Figure.1 shows the basic flow of the
system. To transmit the gesture commands wirelessly ZigBee series2 module is used . Zigbee
coordinator is connected to the serial port of the system through USB Explorer board. Zigbee
router is mounted on the voltage regulator to regulate the voltage to 3.3V. In order to control
the robotic vehicle Arduino board is used

Figure8: Flowchart

15
A. Video From Webcam

The user wears coloured tapes to provide input gestures to the system. The inbuilt webcam of
the computer is used as a sensor to detect the hand movements of the user. It captures the real
time video at a fixed frame rate and resolution which is determined by the hardware of the
camera

B. Colour Detection

Colour plays an important role in image processing. Each image is composed of an array of
M*N pixels with M rows and N columns of pixels. Each pixel contains a certain value for
red, green and blue. Based on the threshold values RGB colours can be differentiated. In
colour detection process the required colour can be detected from an image.

C. Gesture Recognition

Gesture recognition can be seen as a way for computers to begin to understand human body
language, thus building a richer bridge between machines and humans than primitive text
user interfaces or even GUIs (Graphical User Interface). Gesture recognition enables humans
to interface with the robot and interact naturally without any mechanical devices. In gesture
recognition based on the colour detected and its position it is interpreted into gestures. Based
on the corresponding gesture, robotic movement is done.

D. Arduino And Zigbee

In order to control the robotic vehicle according to the input gestures given by the user
Arduino Uno board is used. Serial communication between Arduino and MATLAB is
established wirelessly through ZigBee

6.IMPLEMENTATION

16
A. Acquiring Image

The real time video is given by the user. The video is sliced into images at a particular frame
rate. In order to obtain the intensity information, the acquired image is converted into
grayscale image.

B. RGB to Grayscale Conversion

RGB image is converted into grayscale image. It converts the true colour image RGB to the
grayscale image. A grayscale digital image is an image in which the value of each pixel is a
single sample, that is, it carries only intensity information. As compared to a coloured image,
computational complexity is reduced in a grey scale image. All the necessary operations were
performed after converting the image into grey scale.

C. Extraction of Red Colour

From the RGB image the required colour (red) is extracted by subtracting the image. The red,
green and blue colour object is detected by subtracting the RGB suppressed channel from the
grayscale image. This creates an image which contains the detected object as a patch of grey
surrounded by black space. The conversion to binary is required to find the properties of a
monochromatic image.

D. Grey to Binary Scale Conversion

The grey region of the image obtained after subtraction needs to be converted to a binary
image for finding the region of the detected object. A grayscale image consists of a matrix
containing the values of each pixel. Image thresholding is a simple, yet effective, way of
partitioning an image into a foreground and background. Thus, the resultant image obtained
is a monochromatic image consisting of only black and white colours. The conversion to
binary is required to find the properties of a monochromatic image.

E. Finding Centroid of an Object

17
For the user to control the mouse pointer it is necessary to determine a point whose
coordinates can be sent to the cursor. With these coordinates, the system can perform robotic
movements. The centroid is calculated for the detected region. The output of function is a
matrix consisting of the X (horizontal) and Y (vertical) coordinates of the centroid.

F. Serial Communication between MATLAB and Arduino

The number of centroids is transmitted to ZigBee coordinator via COM port of the system.
ZigBee router present in the wireless network receives data from coordinator and it transmits
to the Arduino.

G. Controlling Robotic Vehicle

The Arduino transmits command to robotic vehicle. Based on the command’s appropriate
movements like forward, reverse, turning left and right are performed.

7.SIMULATION RESULTS
This system is developed in the software MATLAB R2015a under Windows 8.1 operating
platform. In the above software the toolboxes used are Image Acquisition Toolbox, Image
Processing Toolbox, and Instrument Control Toolbox. XCTU software is used for
configuring ZigBee as ZigBee coordinator and ZigBee router. In order to control the robotic
vehicle Arduino Uno is used. The real time gesture is acquired through webcam of the
system. Figure. shows different real time gesture inputs and its corresponding grey scale
image, subtracted image, binary image, centroid plot. According to the number of centroids,
different robotic movements are performed. If the number of centroid is one, the vehicle
moves leftward. If the number of centroid is two, it moves forward. If the number of centroid
is three, it moves rightward. If the number of centroid is four, the vehicle moves backward.
Figure.3 shows the robotic movement based on the input gesture.

18
19
20
 8. APPLICATION

Art:

Drawing application lets the user to draw on any surface by tracking the fingertip
movements of the user’s index finger.

·Zooming:

Like the gestures supported by Multi-Touch based systems, letting the user zoom in, zoom
out or pan using intuitive hand movements.

·Call making and messaging:

One can use the sixth sense to project a keypad onto the hand, then use that virtual keypad to
make a call. Sixth Sense also lets the user draw icons or symbols in the air using the
movement of the index finger and recognizes those symbols as interaction instructions.
Drawing a ‘@’ symbol lets the user check his mail.

As watch:

The gesture of drawing a circle on the user’s wrist projects an analog watch.

·Enhanced news reading:

The Sixth Sense system also augments physical objects the user is interacting with by
projecting more information about these objects projected on them. For example, a
newspaper can show live video news or dynamic information can be provided on a regular
piece of paper.

·Maps:

Drawing a magnifying glass symbol takes the user to the map. The map application lets the

user navigate a map displayed on a nearby surface using hand gestures to zoom and plan.

·Travel:

The system will recognize your boarding pass and let you know whether your flight is on
time and if the gate has changed.

21
·Capturing:

The Sixth Sense system implements a gestural camera that takes photos of the scene the user
is looking at by detecting the ‘framing’ gesture. The user can stop by any surface or wall and
flick through the photos he/she has taken. He can use gestures to sort through photos and
organize and resize them.

Getting information:

Sixth sense uses image recognition or marker technology to recognize products we pick up,
and then feeds us information on those products.

22
 9. CONCLUSION

The gesture-controlled robot system gives an alternative way of controlling robots. It makes
the user to control smart environment by hand gesture interface. This gesture recognition
technique demonstrates the use of MATLAB image processing tools to detect and count the
number of centroids of gestural image. Since this technique makes use of gesture as input
transmitted through ZigBee, it improves the operating range of robot. It eliminates the use of
external hardware supports like remote control in navigation of robot in different direction
and reduces the human efforts in risky environments.

We have discussed about the latest innovative argot that has announces its existence in the
technical field, it recognizes different varieties of actions and gestures, it can easily interact
with other objects. Sixth sense technology will going to bring up a great and wide change in
the way we look our world that’s why this technology must have a bright future. The main
advantage of this technology is that, it is portable as well as interactive than any other device
like laptop, Smartphone, tablets etc. In this report we have also discussed about applications,
disadvantages and advantages of sixth sense technology as well. This new technology has a
wearable device that authorizes us to have the entire world of advanced data readily which is
truly available. The masterstroke of this technology is that it has the potential that holds to
connect INTERNET with real world and superimposing the data on the real-world. The key
point is that it identifies the objects around oneself, let us access the information in that way
we have discussed displays and want that information as well, all these in the simplest way.

23
 REFRENCES:

 Boingboing.net/2009/11/12/sixth-sense-technolo.html
 Pranavmistry.com
 Freshcreation.com/entry/sixth_sense_technology
 [1] Mukti Yadav, Mrinal Yadav and Hemant Sapra , “MATLAB
based Gesture Controlled Robot”, Journal of Basic and Applied
Engineering Research ,Vol3, Issue 2; January-March, 2016.
 AdityaPurkayastha, Akhil Devi Prasad, Arunav Bora,
Akshaykumar Gupta, Pankaj Singh, “Hand Gestures Controlled
Robotic Arm”, Journal of International Academic Research For
Multidisciplinary, Vol2, Issue-4, May 2014.
 Daggu Venkateshwar Rao, Shruti Patil, Naveen Anne Babu and
V.Muthukumar,“Implementation and Evaluation of Image
Processing Algorithms on ReconfigurableArchitecture using
Cbased Hardware Descriptive Languages”, International Journal of
Theoretical and Applied Computer Sciences, Vol1 pp. 9–34, 2006

24