SixthSense
Manisha Sharma
Computer Science
Dronacharya College of Engineering
manisha.s2212@gmail.com
Abstract
SixthSense' is a wearable gestural interface that augments
the physical world around us with digital information and
lets us use natural hand gestures to interact with that
information. It was developed by Pranav Mistry, (PhD
student in the Fluid Interfaces Group at the MIT Media
Lab).By using a camera and a tiny projector mounted in a
pendant like wearable device, 'SixthSense' sees what you
see and visually augments any surfaces or objects we are
interacting with. It projects information onto surfaces,
walls, and physical objects around us, and lets us interact
with the projected information through natural hand
gestures, arm movements, or our interaction with the
object itself. This wearable gestural interface attempts to
free information from its confines by seamlessly
integrating it with reality, and thus making the entire
world your computer.
Introduction
'SixthSense' is a wearable gestural interface that
augments the physical world around us with digital
information and lets us use natural hand gestures to
interact with that information .We've evolved over
millions of years to sense the world around us. When we
encounter something, someone or some place, we use our
five natural senses to perceive information about it; that
information helps us make decisions and chose the right
actions to take. But arguably the most useful information
that can help us make the right decision is not naturally
perceivable with our five senses, namely the data,
information and knowledge that mankind has
accumulated about everything and which is increasingly
all available online. Although the miniaturization of
computing devices allows us to carry computers in our
pockets, keeping us continually connected to the digital
world, there is no link between our digital devices and
our interactions with the physical world. Information is
confined traditionally on paper or digitally on a screen.
SixthSense bridges this gap, bringing intangible, digital
information out into the tangible world, and allowing us
to interact with this information via natural hand gestures.
‘SixthSense’ frees information from its confines by
seamlessly integrating it with reality, and thus making the
entire world your computer.
The SixthSense prototype is comprised of a pocket
projector, a mirror and a camera. The hardware

components are coupled in a pendant like mobile
wearable device. Both the projector and the camera are
connected to the mobile computing device in the user’s
pocket. The projector projects visual information
enabling surfaces, walls and physical objects around us
to be used as interfaces; while the camera recognizes and
tracks user's hand gestures and physical objects using
computer-vision based techniques. The software program
processes the video stream data captured by the camera
and tracks the locations of the colored markers (visual
tracking fiducials) at the tip of the user’s fingers using
simple computer-vision techniques. The movements and
arrangements of these fiducials are interpreted into
gestures that act as interaction instructions for the
projected application interfaces. The maximum number
of tracked fingers is only constrained by the number of
unique fiducials, thus SixthSense also supports multi-
touch and multi-user interaction.
The SixthSense prototype implements several
applications that demonstrate the usefulness, viability and
flexibility of the system. The map application lets the
user navigate a map displayed on a nearby surface using
hand gestures, similar to gestures supported by Multi-
Touch based systems, letting the user zoom in, zoom out
or pan using intuitive hand movements. The drawing
application lets the user draw on any surface by tracking
the fingertip movements of the user’s index finger.
SixthSense also recognizes user’s freehand gestures
(postures). For example, the SixthSense system
implements a gestural camera that takes photos of the
scene the user is looking at by detecting the ‘framing’
gesture.
The user can stop by any surface or wall and flick
through the photos he/she has taken. SixthSense also lets
the user draw icons or symbols in the air using the
movement of the index finger and recognizes those
symbols as interaction instructions.
For example, drawing a magnifying glass symbol takes
the user to the map application or drawing an ‘@’ symbol
lets the user check his mail. The SixthSense system also
augments physical objects the user is interacting with by
projecting more information about these objects projected
on them. For example, a newspaper can show live video
news or dynamic information can be provided on a
regular piece of paper. The gesture of drawing a circle on
the user’s wrist projects an analog watch
.
Basic Components

It consists of certain commonly available components,
which are intrinsic to its functioning. These include a
camera, a portable battery-powered projection system
coupled with a mirror and a cell phone. All these
components communicate to the cell phone, which acts as
the communication and computation device. The entire
hardware apparatus is encompassed in a pendant-shaped
mobile wearable device. Basically the camera recognises
individuals, images, pictures, gestures one makes with
their hands and the projector assists in projecting any
information on whatever type of surface is present in
front of the person. The usage of the mirror is significant
as the projector dangles pointing downwards from the
neck. To bring out variations on a much higher plane, in
the demo video which was broadcasted to showcase the
prototype to the world, Mistry uses coloured caps on his
fingers so that it becomes simpler for the software to
differentiate between the fingers, demanding various
applications. The software program analyses the video
data caught by the camera and also tracks down the
locations of the coloured markers by utilising single
computer vision techniques. One can have any number of
hand gestures and movements as long as they are all
reasonably identified and differentiated for the system to
interpret it, preferably through unique and varied
fiducials. This is possible only because the ‘Sixth Sense’
device supports multi-touch and multi-user interaction.
Hardware Setup
Camera
Captures an object in view and tracks the user’s hand
gestures. It sends the data to smart phone. It acts as a
digital eye, connecting you to the world of digital
information.
Projector
The Projector projects visual information enabling
surfaces and physical objects to be used as interfaces.
The project itself contains a battery inside, with 3 hours
of battery life. A tiny LED projector displays data sent
from the smart phone on any surface in view–object,
wall, or person. Pocket projector Pk101 from Optoma is
used. It is Suitable for mobile usage
Mirror
The usage of the mirror is significant as the projector
dangles pointing downwards from the neck. The mirror is
used to focus projections on surface.
Mobile Component
A Web-enabled smart phone in the user’s pocket
processes the video data. Other software searches the
Web and interprets the hand gestures .Nokia n95 smart
phone is used (running Symbian O.S s60 edition). It has
multitasking capability. Built-in camera provides
execution of both Gesture tracking engine and Gesture
enabled application
Colored Markers
Marking the tip of user’s fingers with red, yellow, green,
and blue tape helps the webcam recognize gestures. The
movements and arrangements of these makers are
interpreted into gestures that act as interaction
instructions for the projected application interfaces.
Software Setup
Applications are implemented using JAVA 2 MICRO
edition, a Java platform designed for embedded systems
where target devices range from industrial controls to
mobile phones.Computer vision library is written in
symbian C++ (used in Gesture tracking).The software for
the sixth sense prototype is developed on a Microsoft
Windows platform using C#, WPF and open CV.The
software works on the basis of computer vision.A small
camera acting as an eye, connecting us to the world of
digital information. Processing is happening in the
mobile phone, and basically works on computer vision
algorithms.Approx 50,000 lines of code are used
Kinds Of Gestures Recognized
MULTI-TOUCH GESTURES are like the ones we see
in the iphone – where we touch the screen and make the
map move by pinching and dragging.
FREEHAND GESTURES are like when you take a
picture or a namaste gesture to start the projection on the
wall. ICONIC GESTURES drawing an icon in the air.
Like, Whenever we draw a star, show us the weather
details. When we draw a magnifying glass, show us the
map. This system is very customizable. We can make our
own gesture which our sixth sense device can understand.
We can change the Sixth Sense to our need.
Applications
Make a call
The Sixth Sense prototype can be used to project a
keypad onto your hand and then use that virtual keypad
to make a call.
Call up a map
With the map application we can call up the map of our
choice and then use thumbs and index fingers to navigate
the map
Time details
The user can draw a circle on your wrist to get a virtual
watch that gives you the correct time
Multimedia reading experiences
Sixth Sense can enrich a user’s multimedia experiences.
It can be programmed to project related videos onto
newspaper articles you are reading.
Drawing applications
The drawing application lets the user draw on any surface
by tracking the fingertip movements of the user’s index
finger
Zooming features
The user can zoom in or zoom out using intuitive hand
movements
Access book information
The system can project Amazon ratings on that book, as
well as reviews and other relevant information
Access product information
Sixth Sense uses image recognition or marker technology
to recognize products we pick up, and then feeds us
information on those products
Flight updates
The system will recognize your boarding pass and let you
know whether your flight is on time and if the gate has
changed
Take pictures
If you fashion your index fingers and thumbs into a
square ("framing" gesture), the system will snap a photo.
After taking the desired number of photos, we can project
them onto a surface, and use gestures to sort through the
photos, and organize and resize them.
Related Technologies
Augmented reality
Augmented reality (AR) is a term for a live direct or
indirect view of a physical real-world environment whose
elements are augmented by virtual computer-generated
sensory input such as sound or graphics. It is related to a
more general concept called mediated reality in which a
view of reality is modified (possibly even diminished
rather than augmented) by a computer. As a result, the
technology functions by enhancing one’s current
perception of reality.
In the case of Augmented Reality, the augmentation is
conventionally in real-time and in semantic context with
environmental elements, such as sports scores on TV
during a match. With the help of advanced AR
technology (e.g. adding computer vision and object
recognition) the information about the surrounding real
world of the user becomes interactive and digitally
usable. Artificial information about the environment and
the objects in it can be stored and retrieved as an
information layer on top of the real world view. The term
augmented reality is believed to have been coined in
1990 by Thomas Caudell, an employee of Boeing at the
time[1].Augmented reality research explores the
application of computer-generated imagery in live-video
streams as a way to expand the real-world. Advanced
research includes use of head-mounted displays and
virtual retinal displays for visualization purposes, and
construction of controlled environments containing any
number of sensors and actuators.
Gesture recognition
Gesture recognition is a topic in computer science and
language technology with the goal of interpreting human
gestures via mathematical algorithms. Gestures can
originate from any bodily motion or state but commonly
originate from the face or hand. Current focuses in the
field include emotion recognition from the face and hand
gesture recognition. Many approaches have been made
using cameras and computer vision algorithms to
interpret sign language. However, the identification and
recognition of posture, gait, proxemics, and human
behaviors is also the subject of gesture recognition
techniques.[1]
Gesture recognition can be seen as a way for computers
to begin to understand human body language, thus
building a richer bridge between machines and humans
than primitive text user interfaces or even GUIs
(graphical user interfaces), which still limit the majority
of input to keyboard and mouse. Gesture recognition
enables humans to interface with the machine (HMI) and
interact naturally without any mechanical devices. Using
the concept of gesture recognition, it is possible to point a
finger at the computer screen so that the cursor will move
accordingly. This could potentially make conventional
input devices such as mouse, keyboards and even touch-
screens redundant. Gesture recognition can be conducted
with techniques from computer vision and image
processing. The literature includes ongoing work in the
computer vision field on capturing gestures or more
general human pose and movements by cameras
connected to a computer.[2][3][4][5]Gesture recognition and
pen computing In some literature, the term gesture
recognition has been used to refer more narrowly to non-
text-input handwriting symbols, such as inking on a
graphics tablet, multi-touch gestures, and mouse gesture
recognition. This is computer interaction through the
drawing of symbols with a pointing device cursor.
Computer vision
Computer vision is the science and technology of
machines that see, where see in this case means that the
machine is able to extract information from an image that
is necessary to solve some task. As a scientific discipline,
computer vision is concerned with the theory behind
artificial systems that extract information from images.
The image data can take many forms, such as video
sequences, views from multiple cameras, or multi-
dimensional data from a medical scanner. As a
technological discipline, computer vision seeks to apply
its theories and models to the construction of computer
vision systems. Examples of applications of computer
vision include systems for: Controlling processes (e.g., an
industrial robot or an autonomous vehicle).,Detecting
events (e.g., for visual surveillance or people counting).,
Organizing information e.g., for indexing databases of
images and image sequences, Modeling objects or
environments e.g., industrial inspection, medical image
analysis and Interaction (e.g., as the input to a device for
computer-human interaction).
Computer vision is closely related to the study of
biological vision. The field of biological vision studies
and models the physiological processes behind visual
perception in humans and other animals. Computer
vision, on the other hand, studies and describes the
processes implemented in software and hardware behind
artificial vision systems. Interdisciplinary exchange
between biological and computer vision has proven
fruitful for both fields. Computer vision is, in some ways,
the inverse of computer graphics. While computer
graphics produces image data from 3D models, computer
vision often produces 3D models from image data. There
is also a trend towards a combination of the two
disciplines, e.g., as explored in augmented reality. Sub-
domains of computer vision include scene reconstruction,
event detection, video tracking, object recognition,
learning, indexing, motion estimation, and image
restoration.
Radio Frequency Identification
Radio frequency identification (RFID) is a generic term
that is used to describe a system that transmits the
identity (in the form of a unique serial number) of an
object or person wirelessly, using radio waves. It is
basically an electronic tagging technology that allows the
detection, tracking of tags and consequently the objects
that they are affixed to. It's grouped under the broad
category of automatic identification technologies.RFID is
in use all around us. If you have ever chipped your pet
with an ID tag, used EZPass through a toll booth, or paid
for gas using SpeedPass, you've used RFID. In addition,
RFID is increasingly used with biometric technologies
for security. Unlike ubiquitous UPC bar-code technology,
RFID technology does not require contact or line of sight
for communication. RFID data can be read through the
human body, clothing and non-metallic materials
Conclusion
Information is often confined to paper or computer
screens. SixthSense frees data from these confines and
seamlessly integrates information and reality. With the
miniaturization of computing devices, we are always
connected to the digital world, but there is no link
between our interactions with these digital devices and
our interactions with the physical world. SixthSense
bridges this gap by augmenting the physical world with
digital information, bringing intangible information into
the tangible world.
Sixth Sense recognizes the objects around us,
displaying information automatically and
letting us to access it in any way we need.The
Sixth Sense prototype implements several
applications that demonstrate the usefulness,
viability and flexibility of the system. Allowing
us to interact with this information via natural
hand gestures. The potential of becoming the
ultimate "transparent" user interface for
accessing information about everything
around us. The SixthSense prototype implements
several applications that demonstrate the usefulness,
viability and flexibility of the system.
References
1 http://www.technologyreview.com/TR35/Profile.asp
x?TRID=816
2 Power point presentationon SIXTH SENSE
TECHNOLOGY by Sandeep.s 4pa05ec091
http://www.scribd.com/doc/30495435/Sixth-Sense-
Technology#open_download
3 http://www.pranavmistry.com/projects/sixthsense/
4 http://news.softpedia.com/news/Next-Gen-039-
Sixth-Sense-039-Device-Created-at-MIT-
103879.shtml
5 http://en.wikipedia.org/wiki/Augmented_reality
6 http://en.wikipedia.org/wiki/Gesture_recognition
7 http://en.wikipedia.org/wiki/Computer_vision
8 http://www.aimglobal.org/technologies/RFID/what_i
s_rfid.asp
9 http://news.bbc.co.uk/2/hi/technology/7997961.stm
10 http://boingboing.net/2009/11/12/sixth-sense-
technolo.html
11 http://gizmodo.com/5167790/sixth-sense-
technology-may-change-how-we-look-at-the-world-
forever
12 http://www.ted.com/talks/pattie_maes_demos_the_si
xth_sense.html
13 http://theviewspaper.net/sixth-sense-technology-will-
revolutionize-the-world/
14 http://www.freshcreation.com/entry/sixth_sense_tech
nology/
15 Dana H. Ballard and Christopher M. Brown (1982).
Computer Vision. Prentice Hall. ISBN 0131653164.
http://homepages.inf.ed.ac.uk/rbf/BOOKS/BANDB/
bandb.htm.
16 David Marr (1982). Vision. W. H. Freeman and
Company. ISBN 0-7167-1284-9.
17 Azriel Rosenfeld and Avinash Kak (1982). Digital
Picture Processing. Academic Press. ISBN 0-12-
597301-2.
18 Berthold Klaus Paul Horn (1986). Robot Vision. MIT
Press. ISBN 0-262-08159-8.
19 Olivier Faugeras (1993). Three-Dimensional
Computer Vision, A Geometric Viewpoint. MIT
Press. ISBN 0-262-06158-9.
20 Tony Lindeberg (1994). Scale-Space Theory in
Computer Vision. Springer. ISBN 0-7923-9418-6.
http://www.nada.kth.se/~tony/book.html.
21 James L. Crowley and Henrik I. Christensen (Eds.)
(1995). Vision as Process. Springer-Verlag. ISBN 3-
540-58143-X and ISBN 0-387-58143-X.
22 Gösta H. Granlund and Hans Knutsson (1995).
Signal Processing for Computer Vision. Kluwer
Academic Publisher. ISBN 0-7923-9530-1.
23 Reinhard Klette, Karsten Schluens and Andreas
Koschan (1998). Computer Vision - Three-
Dimensional Data from Images. Springer, Singapore.
ISBN 981-3083-71-9.
24 Emanuele Trucco and Alessandro Verri (1998).
Introductory Techniques for 3-D Computer Vision.
Prentice Hall. ISBN 0132611082.
25 Milan Sonka, Vaclav Hlavac and Roger Boyle
(1999). Image Processing, Analysis, and Machine
Vision. PWS Publishing. ISBN 0-534-95393-X.
26 Bernd Jähne and Horst Haußecker (2000). Computer
Vision and Applications, A Guide for Students and
Practitioners. Academic Press. ISBN 0-12-379777-
2.
27 Linda G. Shapiro and George C. Stockman (2001).
Computer Vision. Prentice Hall. ISBN 0-13-030796-
3.
28 Bernd Jähne (2002). Digital Image Processing.
Springer. ISBN 3-540-67754-2.
29 David A. Forsyth and Jean Ponce (2003). Computer
Vision, A Modern Approach. Prentice Hall. ISBN 0-
13-085198-1.
30 Richard Hartley and Andrew Zisserman (2003).
Multiple View Geometry in Computer Vision.
Cambridge University Press. ISBN 0-521-54051-8.
31 Gérard Medioni and Sing Bing Kang (2004).
Emerging Topics in Computer Vision. Prentice Hall.
ISBN 0-13-101366-1.
32 Tim Morris (2004). Computer Vision and Image
Processing. Palgrave Macmillan. ISBN 0-333-
99451-5.
33 E. Roy Davies (2005). Machine Vision : Theory,
Algorithms, Practicalities. Morgan Kaufmann.
ISBN 0-12-206093-8.
34 R. Fisher, K Dawson-Howe, A. Fitzgibbon, C.
Robertson, E. Trucco (2005). Dictionary of
Computer Vision and Image Processing. John Wiley.
ISBN 0-470-01526-8.
35 Nikos Paragios and Yunmei Chen and Olivier
Faugeras (2005). Handbook of Mathematical Models
in Computer Vision. Springer. ISBN 0-387-26371-3.
36 Wilhelm Burger and Mark J. Burge (2007). Digital
Image Processing: An Algorithmic Approach Using
Java. Springer. ISBN 1846283795 and ISBN
3540309403. http://www.imagingbook.com/.
37 Pedram Azad, Tilo Gockel, Rüdiger Dillmann
(2008). Computer Vision - Principles and Practice.
Elektor International Media BV. ISBN 0905705718.
http://ivt.sourceforge.net/book.html.