Journal of Sci. Engg. & Tech. Mgt.

Vol 2 (2), July 2010

“Multi Touch Technology”

The Burgeoning Mean of Human Computer Interface
1
H.B.Kekre, 2Sudeep D. Thepade, 3Digvijay Singh, 4Tushant Mitaal, 5Aman Pratap Singh
1
Senior Professor, 2Associate Proffesor, 3,4,5B.Tech. Student
Computer Engineering Department,
MPSTME, SVKM’s NMIMS University, Mumbai

Abstract
Multi-touch (or multi-touch) denotes a set of interaction techniques which allow computer or
mobile users to control graphical user interface with more than one finger at either application or
system level. The paper takes quick review of Touch screen technology for multi touch displays.
Further the hardware constitution details are discussed. Software support along with applications
are also discussed with some future directions.

opposed to the single touch screen (e.g.

I. INTRODUCTION
Multi-touch (or multitouch) denotes a
set of interaction techniques which allow
computer or mobile users to control graphical
user interface with more than one finger at
either application or system level [1]. An
interaction technique, user interface technique
or input technique is a combination of
hardware and software elements that provides a
way for computer users to accomplish a single
task. For example, one can go back to the
previously visited page on a Web browser by
either clicking a button, pressing a key,
performing a mouse gesture or uttering a
speech command. It is a key concept in human-
computer interaction.
Fig.1.1. Multi-touch Screen [1]
While touch sensing is commonplace
for single points of contact, multi-touch
sensing enables a user to interact with a system
with more than one finger at a time, as in
chording and bi-manual operations. Such
sensing devices are inherently also able to
accommodate multiple users simultaneously,
which is especially useful for larger interaction
scenarios such as interactive walls and
tabletops. Multi-touch consists of a touch
screen (screen, overlay, table, wall, etc.) or
touchpad, as well as software that recognizes
multiple simultaneous touch points, as
computer touchpad, ATM), which recognizes
only one touch point. This effect is achieved
through a variety of means, including: heat,
finger pressure, high capture rate cameras,
infrared light, optic capture, tuned
electromagnetic induction, ultrasonic
receivers, transducer microphones, laser
rangefinders, and shadow capture [2].
Many products using multi-touch
interfaces exist and are being developed.
Multi-touch is used on portable devices
including the Apple iPhone, HTC Hero,
Microsoft's Zune HD, and the Palm Pre, as
well as desktop products such as the Microsoft
Surface and the DELL Latitude XT2.
Hobbyists have developed extremely low cost
touch screen technologies using webcams,
projection of image, and software that senses
touch patterns via the webcam. Multi touch
screens larger than 50" are possible at costs
under $100 to $400 depending on the projector
& equipment used. DIY multi-touch screens of
extremely large size (wall sized or larger) are
also quite possible and very cheaply
constructed and posted in numerous places on
the web.
Fig.1.2. View of Multi-touch Screens [28]
II. HISTORY

25

Multi-touch technology dates back to
1982, when Nimish Mehta at the University of
Toronto developed the first finger pressure
multi-touch display [2]. In 1983, Bell Labs at
Murray Hill published a comprehensive
discussion of touch-screen based interfaces.
[27] In 1984 Bell Labs engineered a touch
screen that could change images with more
than one hand. The group at the University of
Toronto stopped working on hardware and
moved on to software and interfaces,
expecting that they would have access to the
Bell Labs work. Mainstream exposure to
multi-touch technology occurred in the year
2007, when Apple unveiled the iPhone and
Microsoft debuted surface computing. The
iPhone in particular has spawned a wave of
interest in multi-touch computing
III. Conventional Touch Screens
Touchscreen is a display that can
detect the presence and location of a touch
within the display area. The term generally
refers to touch or contact to the display of the
device by a finger or hand. Touchscreens can
also sense other passive objects, such as a
stylus. However, if the object sensed is active,
as with a light pen, the term touchscreen is
generally not applicable. The ability to interact
directly with a display typically indicates the
presence of a touchscreen. The touchscreen
has two main attributes. First, it enables one to
interact with what is displayed directly on the
screen, where it is displayed, rather than
indirectly with a mouse or touchpad. Secondly,
it lets one do so without requiring any
intermediate device, again, such as a stylus
that needs to be held in the hand. Such
displays can be attached to computers or, as
terminals, to networks. They also play a
prominent role in the design of digital
appliances such as the personal digital
assistant (PDA), satellite navigation devices,
mobile phones, and video games.
There are a number of types of
touchscreen technologoies, some of them are
as listed below..
A. Resistive Touch Screen [4] [3]
B. Surface acoustic wave [5] [3]
C. Capacitive Touch Screen [3]
D. Surface capacitance [6] [3]
E. Infrared Touch Screen [3]
F. Strain gauge [8], [9], [10].
G. Optical imaging[3]
26
Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010
H. Dispersive signal technology [11] [3].
I. Acoustic pulse recognition [12] [3].
IV. Multi-touch displays
A. Perceptive Pixel[13][1]
Fig.4.1. Jefferson Y. Han debuting his initial
multi-touch system at TED [1]
Perceptive Pixel is a company founded
by New York University consulting research
scientist Jefferson Y. Han that creates wall
displays and tables. The displays use infrared
light emitting diodes along with an infrared
camera to determine the point of contact. Han
envisions large collaborative spaces that will
allow multiple users to work and interact.
Perceptive Pixel’s technology is currently
being utilized, in the form of the Multi-Touch
Collaboration Wall, by CNN and an
unspecified government contractor. But its 6-
figure price even scared most interested
medium business away.
B. Microsoft Surface[14][15]
In 2001 Steve Bathiche and Andy
Wilson of Microsoft began work on an idea for
an interactive table that mixes both physical
and virtual worlds.
Fig.4.2. Microsoft Surface [1]
Research and Development expanded
rapidly in 2004, once the idea caught the
attention of Microsoft Chairman Bill Gates. In
2007 Microsoft introduced Microsoft Surface,
a functional multi-touch table-top computer
based on a standard PC platform including an
Intel Core 2 Duo processor, Windows Vista,
and 2 GB of RAM. Essentially, Microsoft
Surface is a computer embedded in a table
with a large, flat, touch-responsive display on
the top. The table uses small cameras (as

opposed to finger pressure or heat) that enable
it to react to the touch of any object. The unit
has eight different modes that allow users to
perform an array of activities, ranging from
organizing pictures and videos to ordering
food at a restaurant. Multiple users have the
ability to work on the table at one time. The
preliminary launch was on April 17, 2008,
when Surface became available for customer
use in AT&T stores. Although the product is
not available for widespread purchase, it can
possibly be bought directly from Microsoft for
$13,500.
C. 3M Multi-touch[1]
3M is a company known for many
innovations in the touch field, including
overlays incorporated in other vendor's
products. 3M provides a Multi-touch
developer kit which includes a projected
capacitive glass surfaced 19" monitor with
stand and a software development kit which
includes support for the upcoming Windows 7
operating system. The unique features at this
price point are 10 finger multi-touch and 15ms
touch response with full edge support and
finger identification when both are touching.
D. Tyco Electronics: Elo Touch Systems
Tyco Electronics is a electronics
component manufacturer which also sells
touch technology through their Elo Touch
Systems division. Elo Touch Systems provides
integrator components, custom solutions, kits,
and fully integrated touch monitors through
resellers. Elo Touch Systems provides multiple
technologies for single touch screens: Acoustic
Pulse Recognition (APR); IntelliTouch
Surface Acoustic Wave (SAW); and
CarrollTouch InfraRed (IR). For Multi-touch
they have been an early integrator working on
Microsoft Windows 7, initially announcing
support for 2 finger multi-touch support with
IntelliTouch Surface Acoustic Wave and
technology built on CarrollTouch IR touch
technologies, in both 19" and 22" form factors.
V. HARDWARE CONSTITUTION
Multi-touch (or multi-touch) denotes a
set of interaction techniques that allow
computer users to control graphical
applications with several fingers. Multi-touch
devices consist of a touch screen (e.g.,
computer display, table, wall) or touchpad, as
well as software that recognizes multiple
simultaneous touch points, as opposed to the
27
Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010
standard touch-screen (e.g. computer
touchpad, ATM), which recognizes only one
touch point.
The Natural User Interface and its
influence on multi-touch gestural interface
design has brought key changes in computing
hardware design, especially the creation of
“true” multi-touch hardware systems (i.e.
support for more than two inputs). The aim of
NUI Group is to provide an open platform
where hardware and software knowledge can
be exchanged freely; through this free
exchange of knowledge and information, there
has been an increase in development in regards
to hardware. On the hardware frontier, NUI
Group aims to be an informational resource
hub for others interested in prototyping and/or
constructing, a low cost, high resolution open-
source multi-input hardware system. Through
the community research efforts, there have
been improvements to existing multi-touch
systems as well as the creation of new
techniques that allow for the development of
not only multi-touch hardware systems, but
also multi-modal devices. At the moment there
are five major techniques being refined by the
community that allow for the creation of a
stable multi-touch hardware systems; these
include: Jeff Han’s pioneering Frustrated Total
Internal Reflection (FTIR), Rear Diffused
Illumination (Rear DI) such as Microsoft’s
Surface Table, Laser Light Plan (LLP)
pioneered in the community by Alex Popovich
and also seen in Microsoft’s LaserTouch
prototype, LED-Light Plane (LED-LP)
developed within the community by Nima
Motamedi, and finally Diffused Surface
Illumination (DSI) developed within the com-
munity by Tim Roth. These five techniques
being utilized by the community work on the
principal of Computer Vision and optics
(cameras). While optical sensing makes up the
vast majority of techniques in the NUI Group
community, there are several other sensing
techniques that can be utilized in making
natural user interface and multitouch devices.
Some of these sensing devices include
proximity, acoustic, capacitive, resistive,
motion, orientation, and pressure. Often,
various sensors are combined to form a
particular multitouch sensing technique. In this
chapter, we explore some of the mentioned
techniques. Optical or light sensing (camera)
based solutions make up a large percentage of
multi-touch devices. The scalability, low cost

and ease of setup are suggestive reasoning for
the popularity of optical solutions. Stereo
Vision, Overhead cameras, Frustrated Total
Interal Reflection, Front and Rear Diffused
Illumination, Laser Light Plane, and Diffused
Surface Illumination are all examples of
camera based multi-touch systems. Each of
these techniques consist of an optical sensor
(typically a camera), infrared light source, and
visual feedback in the form of projection or
LCD. Prior to learning about each particular
technique, it is important to understand these
three parts that all optical techniques share.
A. Introduction to Optical Multi-Touch
Technologies [16]
1. Infrared Light Sources
Infrared light is mainly used to
distinguish between a visual image on the
touch surface and the object(s)/finger(s) being
tracked. Since most systems have a visual
feedback system where an image from a
projector, LCD or other display is on the touch
surface, it is important that the camera does
not see this image when attempting to track
objects overlyaed on the display. In order to
separate the objects being tracked from the
visual display, a camera, as explained above, is
modified to only see the infrared spectrum of
light; this cuts out the visual image (visible
light spectrum) from being seen by the camera
and therefore, the camera is able to see only
the infrared light that illuminates the
object(s)/finger(s) on the touch surface.
2. Infrared Cameras
For multitouch setups they need to be
modified first. Regular webcams and cameras
block out infrared light, letting only visible
light in. We need just the opposite. Typically,
by opening the camera up, you can simply pop
the filter off, but on expensive cameras this
filter is usually applied directly to the lens and
cannot be modified.
Resolution: The resolution of the camera is
very important. The higher the resolution the
more pixels are available to detect finger or
objects in the camera image. This is very
important for the precision of the touch device.
For small multi-touch surfaces a low
resolution webcam (320 x 240 pixels) can be
sufficient. Larger surfaces require cameras
with a resolution of 640x480 or higher in order
to maintain the precision.
Frame rate: In order to cope with fast
movements and responsiveness of the system a
camera with at least a frame rate of 30 frames
28
Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010
per second (FPS) is recommended. Higher
frame rates provide a smoother and more
responsive experience.
Interface: Basically there are two types of
interfaces that can be used to connect a camera
device to a computer. An IEEE 1394 device is
recommend because it usually has less
overhead and lower latency in transferring the
camera image to the computer. Again, lower
latency results in a more responsive system.
Lens type: Most consumer webcams contain
an infrared (IR) filter that prevents IR light
from reaching the camera sensor. This is done
to prevent image distortion. However for our
purpose, we want to capture and use IR light.
Choosing the right lens can be a difficult task,
fortunately many manufactures provide an
online lens calculator. The calculator
calculates the required focal length based on
two input parameters which are the distance
between the lens and the object (touch surface)
and the width or height of the touch surface.
Be sure to check if the calculator chooses a
proper lens. Lenses with a low focal length
often suffer from severe image distortion
(Barrel distortion / fish eye), which can
complicate the calibration of the touch
tracking software.
For Multi-touch Firewire cameras have
some benefits over normal USB webcams:
• Higher framerate
• Capture size
• Higher bandwidth
• Less overhead for driver (due to less
compression)
B. Frustrated Total Internal Reflection
(FTIR)[16]
When this happens, no refraction
occurs in the material, and the light beam is
totally reflected. Han’s method uses this to
great effect, flooding the inside of a piece of
acrylic with infrared light by trapping the light
rays within the acrylic using the principle of
Total Internal Reflection. When the user
comes into contact with the surface, the light
rays are said to be frustrated, since they can
now pass through into the contact material
(usually skin), and the reflection is no longer
total at that point. [Fig.5.1.] This frustrated
light is scattered downwards towards an
infrared webcam, capable of picking these
‘blobs’ up, and relaying them to tracking
software.

Fig.5.1. Schematic diagram depicting the
bare minimum of parts needed for a FTIR
setup [16]
1. FTIR Layers
Acrylic
According to the paper of Han, it is necessary
to use acrylic for the screen. The minimum
thickness is 6 mm however large screens
should use 1 cm to prevent the screen from
bending.
Before a sheet of acrylic can be used for a
multi-touch screen it needs to be prepared.
Because acrylic often gets cut up roughly, it is
required to polish the sides of the sheet. This is
done to improve the illumination.
Baffle
The baffle is required to hide the light that is
leaking from the sides of the LEDs. This can
be a border of any material (wood/metal).
Diffuser
Without a diffuser the camera will not only see
the touches, but also all objects behind the
surface. By using a diffuser, only bright
objects (touches) will be visible to the camera.
All other ‘noise data’ will be left out.
Compliant layer
With a basic FTIR setup, the performance
mainly depends on how greasy the fingertips
of the user are. Wet fingers are able to make
better contact with the surface. Dry fingers and
objects won’t be able to frustrate the TIR. To
overcome this problem it is recommended to
add a ‘compliant layer’ on top of the surface.
Instead of frustrating the total internal
reflection by touch, a compliant layer will act
as a proxy. The complaint layer can be made
out of a silicon material such as ELASTOSIL
M 4641. To protect and improve the touch
surface, rear projection material such as Rosco
Gray #02105 can be used. With this setup it is
no longer required to have a diffuser on the
rear side.
2. Diffused Illumination (DI)
29
Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010
Diffused Illumination (DI) comes in two main
forms: Front Diffused Illumination and Rear
Diffused Illumination. Both techniques rely on
the same basic principles - the contrast
between the silent image and the finger that
touches the surface.
Front Diffused Illumination
Visible light (often from the ambient
surroundings) is shined at the screen from
above the touch surface. A diffuser is placed
on top or on bottom of the touch surface.
When an object touches the surface, a shadow
is created in the position of the object. The
camera senses this shadow.
Rear Diffused Illumination
Infrared light is shined at the screen from
below the touch surface. A diffuser is placed
on top or on bottom of the touch surface.
When an object touches the surface it reflects
more light than the diffuser or objects in the
background; the extra light is sensed by a
camera. [Fig.5.2.]Depending on the diffuser,
this method can also detect hover and objects
placed on the surface. Rear DI, as
demonstrated in the figure below, requires
infrared illuminators to function.
Fig5.2. Rear DI schematic [16]
C. Laser light plane (LLP)
Infrared lasers are an easy and usually
inexpensive way to create a MT setup using
the LLP method. Most setups go with 2-4
lasers, postioned on the corners of the touch
surface. The laser wattage power rating
(mW,W) is related to the brightness of the
laser, so the more power the brighter the IR
plane will be. The common light wavelengths
used are 780nm and 940nm as those are the
wavelengths available on the Aixiz.com
website where most people buy their laser
modules. Laser modules need to have line
lenses on them to create a light plane. The 120
degree line lens is most commonly used, so as

to reduce the number of lasers necessary to
cover the entire touch surface. Safety when
using lasers of any power is important, so
exercise common sense and be mindful of
where the laser beams are travelling.
Fig5.3. LLP schematic [16]
D. Diffused Surface Illumination (DSI)
DSI uses a special acrylic to distribute
the IR evenly across the surface. Basically use
your standard FTIR setup with an LED Frame
(no compliant silicone surface needed), and
just switch to a special acrylic. [Fig.5.4.]
Fig5.4. DSI Schematic [16]
This acrylic uses small particles that
are inside the material, acting like thousands of
small mirrors. When you shine IR light into
the edges of this material, the light gets
redirected and spread to the surface of the
acrylic. The effect is similar to DI, but with
even illumination, no hotspots, and same setup
process as FTIR. Evonic manufactures some
different types of Endlighten. These vary in
their thickness and also in the amount of
particles inside the material. Available thick-
ness ranges between 6-10mm, follwing “L”,
“XL” and “XXL” for particle amount. The 6
mm (L) is too flexible for a table setup, but the
10 mm (XXL) works nicely.
E. LED Light Plane (LED-LP)
LED-LP is setup the same way as an
FTIR setup except that the thick acrylic that
the infrared light travels through is removed
30
Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010
and the light travels over the touch surface.
This picture [Fig.5.5.] shows the layers that are
common in an LED-LP setup.
Fig5.5. LED-LP 3D Schematic created in
SecondLife [16]
The infrared LEDs are placed around
the touch surface; with all sides being sur-
rounding preferred to get a more even
distribution of light. Similar to LLP, LED-LP
creates a plane of IR light that lays over the
touch surface. Since the light coming from the
LEDs is conical instead of a flat laser plane,
the light will light up objects placed above the
touch surface instead of touching it. This can
be adjusted for by adjusting filter settings in
the software (touchlib/Community Core
Vision) such as the threshold levels to only
pick up objects that are lit up when they are
very close to the touch surface.
A touch surface is a strong, durable
surface that can take the pressure of user
interaction that is optically clear, should be
used. This is usually acrylic or glass. If using
in a projector setup, the image is stopped and
displayed on the projection layer. If using in
an LCD setup, the diffuser is placed below the
LCD screen to evenly distribute the light from
the LCD backlight. The source of infrared
light for an LED-LP setup comes from
infrared LEDs that are placed around at least 2
sides of the acrylic right above the touch
surface. Typically the more sides surrounded,
the better the setup will be in IR prevalent
lighting conditions. Refer to the LED section
for more information on IR LEDs. A computer
webcam is placed on the opposite site of the
touch surface so that is can see the blobs. See
the camera section for more information on
cameras that are commonly used.
VI. SOFTWARE SUPPORT[16]

A. Introduction to Software Programming
Programming for multi-touch input is
much like any other form of coding; however
there are certain protocols, methods, and
standards in the multi-touch world of
programming. Through the work of NUI
Group and other organizations, frameworks
have been developed for several languages,
such as Action Script 3, Python, C, C++, C#,
and Java. Multi-touch programming is two-
fold: reading and translating the “blob” input
from the camera or other input device, and
relaying this information through pre-defined
protocols to frameworks which allow this raw
blob data to be assembled into gestures that
high-level language can then use to interact
with an application. TUIO (Tangible User
Interface Protocol) has become the industry
standard for tracking blob data, and the
following chapters discuss both aspects of
multi-touch software: touch tracking, as well
as the applications operating off of the
tracking frameworks.
B. Tracking
Object tracking has been a cheap yet reliable multi-touch hardware, it
fundamental research element in the field of
Computer Vision. The task of tracking consists
of reliably being able to re-identify a given
object for a series of video frames containing
that object (estimating the states of physical
objects in time from a series of unreliable
observations). In general this is a very difficult
problem since the object needs to first be
detected (quite often in clutter, with occlusion,
or under varying lighting conditions) in all the
frames and then the data must be associated
somehow between frames in order to identify a
recognized object.
1. Tracking for Multi-Touch
Tracking is very important to multi-
touch technology. It is what enables multiple
fingers to perform various actions without
interrupting each other. We are also able to
identify gestures because the trajectory of each
finger can be traced over time, impossible
without tracking. Thankfully, today’s multi-
touch hardware greatly simplifies the task of
tracking an object, so even the simplest filter
in actuality becomes unnecessary. In fact
much of the performance bottleneck of
tracking systems tends to come from gen-
erating and maintaining a model of the
background. Computational costs of these
systems heavily constrict CPU usage unless
31
Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010
clever tricks are employed. However, with the
current infrared (IR) approaches in multi-touch
hardware (e.g., FTIR or DI), an adaptive
background model turns out to be overkill.
Due to the fact that the captured images filter
out (non-infrared) light, much of the
background is removed by the hardware.
Given these IR images, it is often sufficient to
simply capture a single static background
image to remove nearly all of the ambient
light. This background image is then
subtracted from all subsequent frames, the
resultant frames have a threshold applied to
them, and we are left with images containing
‘blobs’ of foreground objects (fingers or
surface widgets we wish to track). Unlike
general tracking solutions, we know that from
one frame of video to another (~33ms for
standard refresh rate of 30Hz), a human finger
will travel only a limited distance.
C. Gesture Recognition
The future of Human-Computer-
Interaction is the Natural User Interface which
is now blurring its boundary with the present.
With the advancement in the development of
wouldn’t be long when we can see multi-touch
screens not only in highly erudite labs but also
in study rooms to drawing rooms and maybe
kitchens too. The mouse and the GUI interface
has been one of the main reasons for the huge
penetration of computers in the society.
However the interaction technique is indirect
and recognition based. The Natural User
Interface with multi-touch screens is intuitive,
contextual and evocative. The shift from GUI
to Gesture based interface will further make
computers an integral but unobtrusive part of
our lifestyle. In its broadest sense, “the notion
of gesture is to embrace all kinds of instances
where an individual engages in movements
whose communicative intent is paramount,
manifest, and openly acknowledged” [17]
Communication through gestures has been one
of the oldest form of interaction in human
civilization owing to various psychological
reasons which, however, is beyond the scope
of present discussion. The GUI systems
leverages previous experience and familiarity
with the application, whereas the NUI
interface leverages the human assumptions and
its logical conclusions to present an intuitive
and contextual interface based on gestures.
Thus a gesture based interface is a perfect
candidate for social and collaborative tasks as
 Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

well for applications involving an artistic

touch. The interface is physical, more visible
and with direct manipulation. However, only
preliminary gestures are being used today in
stand-alone applications on the multi-touch
hardware which gives a lot of scope for its
critics. Multi-touch interface requires a new
approach rather than re-implementing the GUI
and WIMP methods with it. The form of the
gestures determines whether the type of
interaction is actually multi-touch or single-
touch multi-user. We will discuss the kind of
new gesture widgets required, development of
gesture recognition modules and the Fig.6.2. Gesture Widgets Example [16]
supporting framework to fully leverage the
utility of multi-touch hardware and develop Gesture widgets should be designed
customizable, easy to use complex multi-touch with creative thinking, proper user feedback
applications. keeping in mind the context of the application
1. Gesture Widgets and the underlying environment. These gesture
Multi-touch based NUI setups provide widgets can then be extended by the
a strong motivation and platform for a gestural application developer to design complex
interface as it is object based ( as opposed to applications. They should also support
WIMP) and hence remove the abstractness customizable user defined gestures.
between the real world and the application.
The goal of the interface should be to realize a
direct manipulation, higher immersion
interface but with tolerance to the lower
accuracy implied with such an interface. The
popular gestures for scaling, rotating and
translating images with two fingers, commonly
referred as manipulation gestures, are good Fig.6.3. New Generation Gesture Widgets
examples of natural gestures [Fig.6.1.] [16]
[Fig.6.2.].
New types of gesture-widgets
[Fig.6.3.][Fig.6.4.] are required to be build to
fully implement the concept of direct
manipulation with the objects (everything is an
object in NUI with which the user can interact)
and a evocative and contextual environment
and not just trying to emulate mouse-clicks
with a gesture.

Fig.6.4. A View of New Gesture Widgets

[16]

2. Gesture Recognition Process

The primary goal of gesture
recognition research is to create a system
Fig.6.1. Gestures [16] which can identify specific human gestures
and use them to convey information or for
device control. In order to ensure accurate
gesture recognition and an intuitive interface a
number of constraints are applied to the model.

32

The multi-touch setup should provide the
capability for more than one user to interact
with it working on independent (but maybe
collaborative) applications and multiple such
setups working in tandem.
Fig6.5. Some Gesture Recognized by Touch
screen [16]
Gestures are defined by the starting point
within the boundary of one context, end point
and the dynamic motion between the start and
end points. With multi-touch input it should
also be able to recognize meaning of
combination of gestures separated in space or
time. The gesture recognition procedure can be
categorized in three sequential processes:
• Detection of Intention: Gestures should
only be interpreted when they are made
within the application window.
• Gesture Segmentation: The same set of
gestures in the same application can map
to several meanings depending on the
context of the touch events. Thus the touch
events should be again patterned into parts
depending on the object of intention.
These patterned data will be sent to the
gesture recognition module.
• Gesture Classification: The gesture
recognition module will work upon the
patterned data to map it to the correct
command.
There are various techniques for
gesture recognition which can be used alone or
in combination like Hidden Markov Models,
Artificial Neural Networks, and Finite State
Machine etc.
D. Development Frameworks
A number of frameworks have been
released and are being developed to help in the
development of multi-touch applications
providing an interface for the management of
touch events in an object-oriented fashion.
However the level of abstraction is still till the
33
Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010
device input management, in the form of touch
events being sent through TUIO protocol
irrespective of the underlying hardware. The
gesture recognition task which will realize the
true potential of multi-touch surfaces, is still
the job of the client. Often, however, some
basic gestures are already included, in
particular those for natural manipulation (e.g.
of photos), but in general these frameworks
aren’t focused on gestural interfaces. They
rather tend to port the GUI and WIMP canons
to a multitouch environment. Gesture and
gesture recognition modules have currently
gained a lot of momentum with the coming up
of the NUI interface. Some of the important
frameworks are:
1.Sparsh-UI [18]
Sparsh-UI, published under LGPL
license, seems to be the first actual multitouch
gesture recognition framework. It can be
connected to a variety of hardware devices and
supports different operating systems,
programming languages and UI frameworks.
Touch messages are retrieved from the
connected devices and then processed for
gesture recognition. Every visual component
of the client interface can be associated to a
specific set of gestures that will be attempted
to be recognized. New gestures and
recognition algorithms can be added to the
default set included in the package.
2. Grafiti
Grafiti is a C# framework built on top
of the Tuio client that manages multi-touch
interactions in table-top interfaces. The
possible use of tangible objects is particularly
contemplated. It is designed to support the use
of third party modules for (specialized) gesture
recognition algorithms. However a set of
modules for the recognition of some basic
gestures is included in this project.
3. NUIFrame
NUIFrame is a C++ framework based
on the above discussed model (currently under
development). It provides a separate gesture
recognition module to which besides the
touch-event sequence, contextual information
regarding the view of the interface is also
provided by the client application. This
ensures that the same gesture on different
objects can result in different operations
depending on the context. It will also support
custom gestures based on user specification for
a particular command. The set of gesture
widgets will also support automatic debugging

by using pattern generation, according to the
gestures supported by it.
4. AME Patterns Library
The AME Patterns library is a new C+
+ pattern recognition library, currently focused
on real-time gesture recognition. It uses
concept-based programming to express gesture
recognition algorithms in a generic fashion.
The library has recently been released under
the GNU General Public License as a part of
AMELiA (the Arts, Media and Engineering
Library Assortment), an open source library
collection. It implements both a traditional
hidden Markov model for gesture recognition,
as well as some reduced-parameter models that
provide reduced training requirements (only 1-
2 examples) and improved run-time
performance while maintaining good
recognition results.
5. Python[19]
Python is a dynamic object-oriented
programming language that can be used for
many kinds of software development. It offers
strong support for integration with other
languages and tools, comes with extensive
standard libraries, and can be learned in a few
days. Many Python programmers report
substantial productivity gains and feel the
language encourages the development of
higher quality, more maintainable code.
In many ways both Python and multi-touch
have emerged for similar reasons of making
computers easier to use. Although as a
programming language Python certainly
requires advanced technical knowledge of
computers, a major goal is to let developers
write code faster and with less hassle. Python’s
dynamic nature, rapid development
capabilities, make it an ideal language for
prototyping and developing multi-touch
interactions or applications quickly.
VII. Multi-touch Applictions
A. Notebooks and tablets[1]
1. HP Touchsmart TX2
Released in December 2008, the
Touchsmart tx2 was touted as the first
consumer notebook and tablet PC with on-
screen multi-touch control.
2. Dell Latitude XT
In July 2008, Dell released multi-
touch touch-screen drivers for the Latitude XT
Tablet, claiming the "industry’s first
convertible tablet with multi-touch
34
Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010
capabilities” [20]. Dell has partnered with
multi-touch technology startup N-trig to
enable multi-touch capabilities for its tablet.
There have been numerous reports of
serious problems with the XT multi-touch
drivers or hardware interface, which, as of
October 2008, remain unresolved by Dell. The
major issue is cantered on the multi-touch
drivers failing when the computer is booted
when USB devices are connected at boot, or
when certain internal media drives are present.
Often, but not always, the issue is resolved if
the computer is booted without the USB
devices and they are connected after boot. A
similar, but not identical, issue may also be
caused by interfering software, such as the
iTunes helped. Issue resolution is complicated
by the possible failure of the driver install
program to function in the presence of
damaged drivers, requiring manual removal of
related files and registry keys. See External
Links for links to user discussions of this
issue. With the introduction of Windows 7
which offers native multi-touch support,
preliminary tests have shown that many of
these issues with the XT have been solved,
together with a much improved N-Trig Win 7
driver.
3. Sharp Mobius
In April 2009, Sharp unveiled the first
laptop with an optical-sensor LCD pad. The
optical-sensor LCD technology combines LCD
and CCD elements within its pixels and can
detect fingers and objects such as a stylus, and
is claimed to be able to scan a business card
placed on top of the screen. Further
improvements to this function are expected to
enable fingerprint authentication in the future.
4. Lenovo Tablets (x200)
Currently, Lenovo's definition of
"Multitouch" implies that the screen responds
to either a finger or a digitizer pen. However,
it is not capable of detecting simultaneous
touch points on the tablet screen nor any forms
of multi-finger gestures, while the Windows 7
already supports simple dual-touch ones.
B. Software[1]
1. Mac OS X Tiger
Apple Mac OS X 10.4, as well as
derivatives such as iPhone OS, support multi-
touch. This extends from the windowing
environment, through to Safari, iLife and other
software.

2. Linux
Various versions of Linux, and
especially those versions used on small
devices such as the Asus Eee PC and Palm
Pre, have multi-touch support.
3. Windows 7
While previous versions of Windows
do not support multi-touch screens, Windows
7 will support multi-touch when released [21].
The operating system is known to have a
multi-touch mapping application, photo
viewing program, and incorporation in Internet
Explorer 8. In January 2009, Microsoft joined
with other investors who invested twenty-four
million dollars in N-Trig Ltd., which plans to
make computer hardware that takes advantage
of Windows 7's multi-touch support.
Fig.7.1. A multi-touch photos application is
demonstrated in a pre-release version of
Windows 7 [1]
4. 22MILES
One of the advantages of 22MILES
multi-touch framework is OS and multi-touch
hardware independent. Their multi-touch
software can run Windows 7, XP, Vista,
Ubuntu Linux and Mac OS X, while it
supports various multi-touch hardware
technology, such as Infrared Matrix/Camera,
Projected Capacitive, SAW (dual-touch),
resistive, FTIR/ DI/ DSI, LED-LP. But unlike
open-source, their proprietary software and
multitouch gestures only target to business,
such as digital signage, Medical Industry, GIS,
Real Estate, and so on.
5. Multitouch for Java (MT4j)
Another multi-touch framework is
MT4j. It is cross-plattform (XP, Vista, and
Linux) and supports various multi-touch
hardware technology based on the TUIO
protocol. MT4j is open source. MT4j supports
application development in 2D and 3D and
many common input formats such as SVG and
3ds. Using OpenGL for most rendering work,
MT4j based applications perform very well in
combination with a hardware-accelerated
graphics card.
35
Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010
C. Phones and Music players:
1. Apple iPhone
The iPhone is an Internet-connected,
multimedia GSM smartphone designed and
marketed by Apple Inc. Because its minimal
hardware interface lacks a physical keyboard,
the multi-touch screen renders a virtual
keyboard when necessary. The iPhone
functions as a camera phone (also including
text messaging and visual voicemail), a
portable media player (equivalent to a video
iPod), and an Internet client (with email, web
browsing, and Wi-Fi connectivity). The first-
generation phone hardware was quad-band
GSM with EDGE; the second generation
added UMTS with 3.6 Mbps HSDPA; the third
generation adds support for 7.2 Mbps HSDPA
downloading but remains limited to 384 Kbps
uploading as Apple had not implemented the
HSPA protocol.
2. Apple iPod Touch[22]
The iPod Touch (trademarked and
marketed as iPod touch) is a portable media
player, personal digital assistant, and Wi-Fi
mobile platform designed and marketed by
Apple Inc. The product was launched on
September 5, 2007 at an event called The Beat
Goes On. The iPod Touch adds the multi-
touch graphical user interface to the iPod line.
It is the first iPod with wireless access to the
iTunes Store, and also has access to Apple's
App Store, enabling content to be purchased
and downloaded directly on the device.
3. Palm Pre[23]
The Palm Pre is a multimedia
smartphone designed and marketed by Palm
with a multi-touch screen and a sliding
keyboard. The phone was launched on June 6,
2009, and is the first to use Palm's new Linux-
based[1] operating system, webOS. The Pre
functions as a camera phone, a portable media
player, a GPS navigator, and an Internet client
(with text messaging, email, web browsing,
and local Wi-Fi connectivity).
4. Palm Pixi[24]
The Palm Pixi is a multimedia
smartphone under development by smartphone
maker Palm. The device is viewed as a
successor to the Palm Centro smartphone and
will be Palm's second webOS device, after the
Palm Pre. The device was officially announced
on September 8, 2009, and is expected to be
released by the end of 2009.
 Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

5. Zune HD[25] touch-based mechanics alone. Multi-touch on

The Zune HD is a Portable Media the iPhone and iPhone 3G is goo for
Player in the Zune product family by mechanical manipulation, but without
Microsoft released on September 15, 2009 in complementary input modalities like voice or
16GB and 32GB capacities.[4] It utilizes a finger-recognition, the current multi-touch
touchscreen interface for navigation and setup falls short of offering truly immersive
includes Wi-Fi for synchronization, access to user interaction. Imagine trying to resize and
the Zune Marketplace and Web browsing. The rotate a picture while changing its color and
Zune HD utilizes the Nvidia Tegra APX 2600 inserting a bit of text. It wouldn’t make sense
chip, allowing it to play 720p video through to resize and rotate the picture using your
the optional HDMI Zune dock on a high- voice. There are size and rotation-degree
definition television. Otherwise, content will parameters that are just more well-suited for
be scaled down to 480x272 pixels on the manipulation through touch-based multi-touch
player's OLED screen. technology – it’s just easier to pinch/stretch
and rotate the picture with your finger than
VIII. FUTURE SCOPE OF MULTI- with voice commands. Likewise, it’s easier to
TOUCH use voice commands to change the colour of
A. Classrooms of the future to have multi- the background picture or add a text string to
touch desks the picture. Apple proposes, in their latest
patent application, that future multi-touch
technologies will integrate almost all
foreseeable input modalities to complement
touch-based multi-touch. Mechanical
manipulations could be assigned to touch-
based commands, while voice commands are
relegated to functions that require choosing an
option from a list.
Fig.8.1. Future Classrooms
Fig
. 8.2
Researchers at Durham University in
.
the UK are working hard to ensure that the
academic sector isn't left out of the multi-touch
craze. The still-unnamed "interactive multi-
touch desk" is a major part of the SynergyNet
framework, an interactive classroom
environment they're building based on a
gaming engine called jMonkey. There's no
word on how soon SynergyNet will change the Image Resizing [26]
world, but when it does the software will be
open source -- allowing anyone to get into the
code and tweak it to their liking. In the
meantime, schools looking for a "high tech
initiative" of their own will have to be content
with throwing laptops at their students.
B. Apple’s vision of future multi-touch[26]
Apple’s vision of future multi-touch
includes voice, sight, and finger-recognition.
Leave it to the pioneers of multi-touch in the
mobile space to apply for a patent on all future
versions of multi-touch. Apple (NSDQ:
AAPL)’s latest patent filing suggests that
Cupertino is working to integrate more than
just touch-based input modalities in to future Fig.8.3. Using Voice command and Multi-
versions of multi-touch. The problem with the Touch Together [26]
current multi-touch setup is that it’s tailored to

36

Furthermore, Apple envisions future
multi-touch setups incorporating finger-
recognition that would allow the user to assign
different functions to each finger – which
would allow for more complex commands to
be issued. Gaze-vector tracking could allow
your future laptop or mobile phone to
recognize which menu you’d like to select and
automatically drill down to the appropriate
category. Then there’s the proposition of using
finger-tracking and force-sensing technologies
that would allow the user to affect different
degrees of control depending on finger-
pressure and position. Combine all that with
accelerometer support and some pixie dust and
you’ve got the makings of a seriously
immersed control environment. We can’t wait
for a future filled with multi multi-touch and
pixie dust!
IX. REFERENCES
[1] http://en.wikipedia.org/wiki/Multi-
touch#cite_note-11 (accessed on 12
October 2009)
[2] Buxton, Bill. 2008. Multi-Touch Systems
that I Have Known and Loved.
http://www.billbuxton.com/multitouchOve
rview.html (accessed on 29 September
2009).
[3] http://en.wikipedia.org/wiki/Touchscreen
(accessed on 12 October 2009)
[4] "Multi-touch comes to resistive
touchscreens",
http://www.umpcportal.com/2009/02/mult
i-touch-comes-to-resistive-touchscreens
(accessed on 29 September 2009)
[5] Patschon, Mark (1988-03-15), Acoustic
touch technology adds a new input
dimension, Computer Design(accessed on
12 October 2009)
[6] Please Touch! Explore The Evolving
World Of Touchscreen Technology".
electronicdesign.com.
http://electronicdesign.com/Articles/Index.
cfm?AD=1&ArticleID=18592. (accessed
on 12 October 2009)
[7] Kable, Robert G. (1986-07-15),
Electrographic Apparatus, United States
Patent 4,600,807, http://rwservices.no-
ip.info:81/pens/biblio86.html#Kable86
(accessed on 12 October 2009)
[8] Minsky,M.R. (1984-07-01). "Manipulating
simulated objects with real-world gestures
37
Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010
using a force and position sensitive
screen". Computer Graphics. (accessed
on 12 October 2009)
[9] Keuling, Christopher (2008-11-03).
"Touchscreens Press Deep Into Consumer
Electronics". ECN Magazine. (accessed
on 12 October 2009)
[10] Sensors help make ticket machines vandal
proof". Engineeringtalk (accessed on 12
October 2009)
[11] Beyers, Tim (2008-02-13). "Innovation
Series: Touchscreen Technology". The
Motley Fool. (accessed on 12 October
2009)
[12] Acoustic Pulse Recognition Touchscreens,
Elo Touch Systems, 1888-07-31
(accessed on 12 October 2009)
[13] http://www.perceptivepixel.com/
(accessed on 12 October 2009)
[14] http://en.wikipedia.org/wiki/Microsoft_Su
rface (accessed on 12 October 2009)
[15] http://www.microsoft.com/surface/
(accessed on 12 October 2009)
[16] http://nuicode.com/projects/wiki-
book/files (accessed on 12 October 2009)
[17] flosc: Flash Open Sound Control
http://www.benchun.net/flosc/ (accessed
on 12 October 2009)
[18] http://code.google.com/p/sparsh-ui/
(accessed on 12 October 2009)
[19] http://www.python.org/ (accessed on 12
October 2009)
[20] Dell Propels Laptop Innovation With First
Multi-Touch Convertible Tablet, Larger
Solid State Drive (accessed on 12 October
2009)
[21] Windows Vista Team Blog. "Microsoft
demonstrates Multi-touch". MSDN Blogs.
Retrieved 2008-05-28. (accessed on 12
October 2009)
[22] http://en.wikipedia.org/wiki/IPod_Touch
(accessed on 12 October 2009)
[23] http://en.wikipedia.org/wiki/Palm_Pre
(accessed on 12 October 2009)
[24] http://en.wikipedia.org/wiki/Palm_Pixi
(accessed on 12 October 2009)
[25] http://en.wikipedia.org/wiki/Zune_HD
(accessed on 12 October 2009)
[26] http://www.intomobile.com/2008/09/05/ap
ples-vision-of-future-multi-touch-
includes-voice-sight-finger-
recognition.html (accessed on 12 October
2009)

[27] Nakatani, L. H., John A Rohrlich (1983).
"Soft Machines: A Philosophy of User-
Computer Interface Design". Proceedings
of the ACM Conference on Human
Factors in Computing Systems (CHI’83):
12-15.
[28] Jeff Han, Multi Touch interaction
Research http://cs.nyu.edu/~jhan/ftirtouch/
(accessed on 13 october 2009)
X. Author Biographies
Dr. H. B. Kekre has received
B.E. (Hons.) in Telecomm.
Engineering. from Jabalpur
University in 1958, M.Tech
(Industrial Electronics) from IIT Bombay in
1960, M.S.Engg. (Electrical Engg.) from
University of Ottawa in 1965 and Ph.D.
(System Identification) from IIT Bombay
in 1970 He has worked as Faculty of
Electrical Engg. and then HOD Computer
Science and Engg. at IIT Bombay. For 13
years he was working as a professor and head
in the Department of Computer Engg. at
Thadomal Shahani Engineering. College,
Mumbai. Now he is Senior Professor at
MPSTME, SVKM’s NMIMS University. He
has guided 17 Ph.Ds, more than 100
M.E./M.Tech and several B.E./ B.Tech
projects. His areas of interest are Digital
Signal processing, Image Processing and
Computer Networking. He has more than 250
papers in National / International Conferences
and Journals to his credit. Presently He is
Fellow of IETE and Life Member of ISTE
Recently six students working under his
guidance have received best paper awards.
Currently ten research scholars are pursuing
Ph.D. program under his guidance.
Sudeep D. Thepade has
Received B.E.(Computer)
degree from North Maharashtra
University with Distinction in
2003. M.E. in Computer Engineering from
University of Mumbai in 2008 with
Distinction, currently Perusing Ph.D. from
SVKM’s NMIMS University, Mumbai. He
38
Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010
has more than 06 years of experience in
teaching and industry. He was Lecturer in
Dept. of Information Technology at
Thadomal Shahani Engineering College,
Bandra(w), Mumbai for nearly 04 years.
Currently working as Assistant Professor in
Computer Engineering at Mukesh Patel
School of Technology Management and
Engineering, SVKM’s NMIMS University,
Vile Parle(w), Mumbai, INDIA. He is
member of International Association of
Engineers (IAENG) and International
Association of Computer Science and
Information Technology (IACSIT),
Singapore. His areas of interest are Image
Processing and Computer Networks. He has
about 45 papers in National/International
Conferences/Journals to his credit with a Best
Paper Award at International Conference
SSPCCIN-2008 and Second Best Paper
Award at ThinkQuest-2009 National Level
paper presentation competition for faculty.
Digvijay Singh is pursuing B.Tech.
(Computer Science) degree from Mukesh
Patel School of Technology Management and
Engineering, SVKM’s NMIMS University,
Mumbai. His interest in Multi Touch
Technology developed during the seminar
report generation as part of third year B.Tech
curricula.
Tushant Mitaal is pursuing B.Tech.
(Computer Science) degree from Mukesh
Patel School of Technology Management and
Engineering, SVKM’s NMIMS University,
Mumbai. His interest in Multi Touch
Technology developed during the seminar
report generation as part of third year B.Tech
curricula.
Aman Pratap Singh is pursuing B.Tech.
(Computer Science) degree from Mukesh
Patel School of Technology Management and
Engineering, SVKM’s NMIMS University,
Mumbai. His interest in Multi Touch
Technology developed during the seminar
report generation as part of third year B.Tech
curricula.