You are on page 1of 14

Journal of Sci. Engg. & Tech. Mgt.

Vol 2 (2), July 2010

Multi Touch Technology The Burgeoning Mean of Human Computer Interface


H.B.Kekre, 2Sudeep D. Thepade, 3Digvijay Singh, 4Tushant Mitaal, 5Aman Pratap Singh 1 Senior Professor, 2Associate Proffesor, 3,4,5B.Tech. Student Computer Engineering Department, MPSTME, SVKMs NMIMS University, Mumbai

Abstract Multi-touch (or multi-touch) denotes a set of interaction techniques which allow computer or mobile users to control graphical user interface with more than one finger at either application or system level. The paper takes quick review of Touch screen technology for multi touch displays. Further the hardware constitution details are discussed. Software support along with applications are also discussed with some future directions. I. INTRODUCTION opposed to the single touch screen (e.g. computer touchpad, ATM), which recognizes only one touch point. This effect is achieved through a variety of means, including: heat, finger pressure, high capture rate cameras, infrared light, optic capture, tuned electromagnetic induction, ultrasonic receivers, transducer microphones, laser rangefinders, and shadow capture [2]. Many products using multi-touch interfaces exist and are being developed. Multi-touch is used on portable devices including the Apple iPhone, HTC Hero, Microsoft's Zune HD, and the Palm Pre, as well as desktop products such as the Microsoft Surface and the DELL Latitude XT2. Hobbyists have developed extremely low cost touch screen technologies using webcams, projection of image, and software that senses touch patterns via the webcam. Multi touch screens larger than 50" are possible at costs under $100 to $400 depending on the projector & equipment used. DIY multi-touch screens of extremely large size (wall sized or larger) are also quite possible and very cheaply constructed and posted in numerous places on the web.

Multi-touch (or multitouch) denotes a set of interaction techniques which allow computer or mobile users to control graphical user interface with more than one finger at either application or system level [1]. An interaction technique, user interface technique or input technique is a combination of hardware and software elements that provides a way for computer users to accomplish a single task. For example, one can go back to the previously visited page on a Web browser by either clicking a button, pressing a key, performing a mouse gesture or uttering a speech command. It is a key concept in humancomputer interaction.

Fig.1.1. Multi-touch Screen [1] While touch sensing is commonplace for single points of contact, multi-touch sensing enables a user to interact with a system with more than one finger at a time, as in chording and bi-manual operations. Such sensing devices are inherently also able to accommodate multiple users simultaneously, which is especially useful for larger interaction scenarios such as interactive walls and tabletops. Multi-touch consists of a touch screen (screen, overlay, table, wall, etc.) or touchpad, as well as software that recognizes multiple simultaneous touch points, as 25

Fig.1.2. View of Multi-touch Screens [28] II. HISTORY

Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

Multi-touch technology dates back to 1982, when Nimish Mehta at the University of Toronto developed the first finger pressure multi-touch display [2]. In 1983, Bell Labs at Murray Hill published a comprehensive discussion of touch-screen based interfaces. [27] In 1984 Bell Labs engineered a touch screen that could change images with more than one hand. The group at the University of Toronto stopped working on hardware and moved on to software and interfaces, expecting that they would have access to the Bell Labs work. Mainstream exposure to multi-touch technology occurred in the year 2007, when Apple unveiled the iPhone and Microsoft debuted surface computing. The iPhone in particular has spawned a wave of interest in multi-touch computing III. Conventional Touch Screens

H. Dispersive signal technology [11] [3]. I. Acoustic pulse recognition [12] [3]. IV. Multi-touch displays

A. Perceptive Pixel[13][1]

Fig.4.1. Jefferson Y. Han debuting his initial multi-touch system at TED [1] Perceptive Pixel is a company founded by New York University consulting research scientist Jefferson Y. Han that creates wall displays and tables. The displays use infrared light emitting diodes along with an infrared camera to determine the point of contact. Han envisions large collaborative spaces that will allow multiple users to work and interact. Perceptive Pixels technology is currently being utilized, in the form of the Multi-Touch Collaboration Wall, by CNN and an unspecified government contractor. But its 6figure price even scared most interested medium business away. B. Microsoft Surface[14][15] In 2001 Steve Bathiche and Andy Wilson of Microsoft began work on an idea for an interactive table that mixes both physical and virtual worlds.

Touchscreen is a display that can detect the presence and location of a touch within the display area. The term generally refers to touch or contact to the display of the device by a finger or hand. Touchscreens can also sense other passive objects, such as a stylus. However, if the object sensed is active, as with a light pen, the term touchscreen is generally not applicable. The ability to interact directly with a display typically indicates the presence of a touchscreen. The touchscreen has two main attributes. First, it enables one to interact with what is displayed directly on the screen, where it is displayed, rather than indirectly with a mouse or touchpad. Secondly, it lets one do so without requiring any intermediate device, again, such as a stylus that needs to be held in the hand. Such displays can be attached to computers or, as terminals, to networks. They also play a prominent role in the design of digital appliances such as the personal digital assistant (PDA), satellite navigation devices, mobile phones, and video games. There are a number of types of touchscreen technologoies, some of them are as listed below.. A. Resistive Touch Screen [4] [3] B. Surface acoustic wave [5] [3] C. Capacitive Touch Screen [3] D. Surface capacitance [6] [3] E. Infrared Touch Screen [3] F. Strain gauge [8], [9], [10]. G. Optical imaging[3] 26

Fig.4.2. Microsoft Surface [1] Research and Development expanded rapidly in 2004, once the idea caught the attention of Microsoft Chairman Bill Gates. In 2007 Microsoft introduced Microsoft Surface, a functional multi-touch table-top computer based on a standard PC platform including an Intel Core 2 Duo processor, Windows Vista, and 2 GB of RAM. Essentially, Microsoft Surface is a computer embedded in a table with a large, flat, touch-responsive display on the top. The table uses small cameras (as

Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

opposed to finger pressure or heat) that enable it to react to the touch of any object. The unit has eight different modes that allow users to perform an array of activities, ranging from organizing pictures and videos to ordering food at a restaurant. Multiple users have the ability to work on the table at one time. The preliminary launch was on April 17, 2008, when Surface became available for customer use in AT&T stores. Although the product is not available for widespread purchase, it can possibly be bought directly from Microsoft for $13,500. C. 3M Multi-touch[1] 3M is a company known for many innovations in the touch field, including overlays incorporated in other vendor's products. 3M provides a Multi-touch developer kit which includes a projected capacitive glass surfaced 19" monitor with stand and a software development kit which includes support for the upcoming Windows 7 operating system. The unique features at this price point are 10 finger multi-touch and 15ms touch response with full edge support and finger identification when both are touching. D. Tyco Electronics: Elo Touch Systems Tyco Electronics is a electronics component manufacturer which also sells touch technology through their Elo Touch Systems division. Elo Touch Systems provides integrator components, custom solutions, kits, and fully integrated touch monitors through resellers. Elo Touch Systems provides multiple technologies for single touch screens: Acoustic Pulse Recognition (APR); IntelliTouch Surface Acoustic Wave (SAW); and CarrollTouch InfraRed (IR). For Multi-touch they have been an early integrator working on Microsoft Windows 7, initially announcing support for 2 finger multi-touch support with IntelliTouch Surface Acoustic Wave and technology built on CarrollTouch IR touch technologies, in both 19" and 22" form factors. V. HARDWARE CONSTITUTION

Multi-touch (or multi-touch) denotes a set of interaction techniques that allow computer users to control graphical applications with several fingers. Multi-touch devices consist of a touch screen (e.g., computer display, table, wall) or touchpad, as well as software that recognizes multiple simultaneous touch points, as opposed to the 27

standard touch-screen (e.g. computer touchpad, ATM), which recognizes only one touch point. The Natural User Interface and its influence on multi-touch gestural interface design has brought key changes in computing hardware design, especially the creation of true multi-touch hardware systems (i.e. support for more than two inputs). The aim of NUI Group is to provide an open platform where hardware and software knowledge can be exchanged freely; through this free exchange of knowledge and information, there has been an increase in development in regards to hardware. On the hardware frontier, NUI Group aims to be an informational resource hub for others interested in prototyping and/or constructing, a low cost, high resolution opensource multi-input hardware system. Through the community research efforts, there have been improvements to existing multi-touch systems as well as the creation of new techniques that allow for the development of not only multi-touch hardware systems, but also multi-modal devices. At the moment there are five major techniques being refined by the community that allow for the creation of a stable multi-touch hardware systems; these include: Jeff Hans pioneering Frustrated Total Internal Reflection (FTIR), Rear Diffused Illumination (Rear DI) such as Microsofts Surface Table, Laser Light Plan (LLP) pioneered in the community by Alex Popovich and also seen in Microsofts LaserTouch prototype, LED-Light Plane (LED-LP) developed within the community by Nima Motamedi, and finally Diffused Surface Illumination (DSI) developed within the community by Tim Roth. These five techniques being utilized by the community work on the principal of Computer Vision and optics (cameras). While optical sensing makes up the vast majority of techniques in the NUI Group community, there are several other sensing techniques that can be utilized in making natural user interface and multitouch devices. Some of these sensing devices include proximity, acoustic, capacitive, resistive, motion, orientation, and pressure. Often, various sensors are combined to form a particular multitouch sensing technique. In this chapter, we explore some of the mentioned techniques. Optical or light sensing (camera) based solutions make up a large percentage of multi-touch devices. The scalability, low cost

Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

and ease of setup are suggestive reasoning for the popularity of optical solutions. Stereo Vision, Overhead cameras, Frustrated Total Interal Reflection, Front and Rear Diffused Illumination, Laser Light Plane, and Diffused Surface Illumination are all examples of camera based multi-touch systems. Each of these techniques consist of an optical sensor (typically a camera), infrared light source, and visual feedback in the form of projection or LCD. Prior to learning about each particular technique, it is important to understand these three parts that all optical techniques share. A. Introduction to Optical Multi-Touch Technologies [16] 1. Infrared Light Sources Infrared light is mainly used to distinguish between a visual image on the touch surface and the object(s)/finger(s) being tracked. Since most systems have a visual feedback system where an image from a projector, LCD or other display is on the touch surface, it is important that the camera does not see this image when attempting to track objects overlyaed on the display. In order to separate the objects being tracked from the visual display, a camera, as explained above, is modified to only see the infrared spectrum of light; this cuts out the visual image (visible light spectrum) from being seen by the camera and therefore, the camera is able to see only the infrared light that illuminates the object(s)/finger(s) on the touch surface. 2. Infrared Cameras For multitouch setups they need to be modified first. Regular webcams and cameras block out infrared light, letting only visible light in. We need just the opposite. Typically, by opening the camera up, you can simply pop the filter off, but on expensive cameras this filter is usually applied directly to the lens and cannot be modified. Resolution: The resolution of the camera is very important. The higher the resolution the more pixels are available to detect finger or objects in the camera image. This is very important for the precision of the touch device. For small multi-touch surfaces a low resolution webcam (320 x 240 pixels) can be sufficient. Larger surfaces require cameras with a resolution of 640x480 or higher in order to maintain the precision. Frame rate: In order to cope with fast movements and responsiveness of the system a camera with at least a frame rate of 30 frames 28

per second (FPS) is recommended. Higher frame rates provide a smoother and more responsive experience. Interface: Basically there are two types of interfaces that can be used to connect a camera device to a computer. An IEEE 1394 device is recommend because it usually has less overhead and lower latency in transferring the camera image to the computer. Again, lower latency results in a more responsive system. Lens type: Most consumer webcams contain an infrared (IR) filter that prevents IR light from reaching the camera sensor. This is done to prevent image distortion. However for our purpose, we want to capture and use IR light. Choosing the right lens can be a difficult task, fortunately many manufactures provide an online lens calculator. The calculator calculates the required focal length based on two input parameters which are the distance between the lens and the object (touch surface) and the width or height of the touch surface. Be sure to check if the calculator chooses a proper lens. Lenses with a low focal length often suffer from severe image distortion (Barrel distortion / fish eye), which can complicate the calibration of the touch tracking software. For Multi-touch Firewire cameras have some benefits over normal USB webcams: Higher framerate Capture size Higher bandwidth Less overhead for driver (due to less compression) B. Frustrated Total Internal Reflection (FTIR)[16] When this happens, no refraction occurs in the material, and the light beam is totally reflected. Hans method uses this to great effect, flooding the inside of a piece of acrylic with infrared light by trapping the light rays within the acrylic using the principle of Total Internal Reflection. When the user comes into contact with the surface, the light rays are said to be frustrated, since they can now pass through into the contact material (usually skin), and the reflection is no longer total at that point. [Fig.5.1.] This frustrated light is scattered downwards towards an infrared webcam, capable of picking these blobs up, and relaying them to tracking software.

Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

Fig.5.1. Schematic diagram depicting the bare minimum of parts needed for a FTIR setup [16] 1. FTIR Layers Acrylic According to the paper of Han, it is necessary to use acrylic for the screen. The minimum thickness is 6 mm however large screens should use 1 cm to prevent the screen from bending. Before a sheet of acrylic can be used for a multi-touch screen it needs to be prepared. Because acrylic often gets cut up roughly, it is required to polish the sides of the sheet. This is done to improve the illumination. Baffle The baffle is required to hide the light that is leaking from the sides of the LEDs. This can be a border of any material (wood/metal). Diffuser Without a diffuser the camera will not only see the touches, but also all objects behind the surface. By using a diffuser, only bright objects (touches) will be visible to the camera. All other noise data will be left out. Compliant layer With a basic FTIR setup, the performance mainly depends on how greasy the fingertips of the user are. Wet fingers are able to make better contact with the surface. Dry fingers and objects wont be able to frustrate the TIR. To overcome this problem it is recommended to add a compliant layer on top of the surface. Instead of frustrating the total internal reflection by touch, a compliant layer will act as a proxy. The complaint layer can be made out of a silicon material such as ELASTOSIL M 4641. To protect and improve the touch surface, rear projection material such as Rosco Gray #02105 can be used. With this setup it is no longer required to have a diffuser on the rear side. 2. Diffused Illumination (DI)

Diffused Illumination (DI) comes in two main forms: Front Diffused Illumination and Rear Diffused Illumination. Both techniques rely on the same basic principles - the contrast between the silent image and the finger that touches the surface. Front Diffused Illumination Visible light (often from the ambient surroundings) is shined at the screen from above the touch surface. A diffuser is placed on top or on bottom of the touch surface. When an object touches the surface, a shadow is created in the position of the object. The camera senses this shadow. Rear Diffused Illumination Infrared light is shined at the screen from below the touch surface. A diffuser is placed on top or on bottom of the touch surface. When an object touches the surface it reflects more light than the diffuser or objects in the background; the extra light is sensed by a camera. [Fig.5.2.]Depending on the diffuser, this method can also detect hover and objects placed on the surface. Rear DI, as demonstrated in the figure below, requires infrared illuminators to function.

Fig5.2. Rear DI schematic [16] C. Laser light plane (LLP) Infrared lasers are an easy and usually inexpensive way to create a MT setup using the LLP method. Most setups go with 2-4 lasers, postioned on the corners of the touch surface. The laser wattage power rating (mW,W) is related to the brightness of the laser, so the more power the brighter the IR plane will be. The common light wavelengths used are 780nm and 940nm as those are the wavelengths available on the website where most people buy their laser modules. Laser modules need to have line lenses on them to create a light plane. The 120 degree line lens is most commonly used, so as


Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

to reduce the number of lasers necessary to cover the entire touch surface. Safety when using lasers of any power is important, so exercise common sense and be mindful of where the laser beams are travelling.

and the light travels over the touch surface. This picture [Fig.5.5.] shows the layers that are common in an LED-LP setup.

Fig5.3. LLP schematic [16] D. Diffused Surface Illumination (DSI) DSI uses a special acrylic to distribute the IR evenly across the surface. Basically use your standard FTIR setup with an LED Frame (no compliant silicone surface needed), and just switch to a special acrylic. [Fig.5.4.] Fig5.5. LED-LP 3D Schematic created in SecondLife [16] The infrared LEDs are placed around the touch surface; with all sides being surrounding preferred to get a more even distribution of light. Similar to LLP, LED-LP creates a plane of IR light that lays over the touch surface. Since the light coming from the LEDs is conical instead of a flat laser plane, the light will light up objects placed above the touch surface instead of touching it. This can be adjusted for by adjusting filter settings in the software (touchlib/Community Core Vision) such as the threshold levels to only pick up objects that are lit up when they are very close to the touch surface. A touch surface is a strong, durable surface that can take the pressure of user interaction that is optically clear, should be used. This is usually acrylic or glass. If using in a projector setup, the image is stopped and displayed on the projection layer. If using in an LCD setup, the diffuser is placed below the LCD screen to evenly distribute the light from the LCD backlight. The source of infrared light for an LED-LP setup comes from infrared LEDs that are placed around at least 2 sides of the acrylic right above the touch surface. Typically the more sides surrounded, the better the setup will be in IR prevalent lighting conditions. Refer to the LED section for more information on IR LEDs. A computer webcam is placed on the opposite site of the touch surface so that is can see the blobs. See the camera section for more information on cameras that are commonly used. VI. 30 SOFTWARE SUPPORT[16]

Fig5.4. DSI Schematic [16] This acrylic uses small particles that are inside the material, acting like thousands of small mirrors. When you shine IR light into the edges of this material, the light gets redirected and spread to the surface of the acrylic. The effect is similar to DI, but with even illumination, no hotspots, and same setup process as FTIR. Evonic manufactures some different types of Endlighten. These vary in their thickness and also in the amount of particles inside the material. Available thickness ranges between 6-10mm, follwing L, XL and XXL for particle amount. The 6 mm (L) is too flexible for a table setup, but the 10 mm (XXL) works nicely. E. LED Light Plane (LED-LP) LED-LP is setup the same way as an FTIR setup except that the thick acrylic that the infrared light travels through is removed

Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

A. Introduction to Software Programming Programming for multi-touch input is much like any other form of coding; however there are certain protocols, methods, and standards in the multi-touch world of programming. Through the work of NUI Group and other organizations, frameworks have been developed for several languages, such as Action Script 3, Python, C, C++, C#, and Java. Multi-touch programming is twofold: reading and translating the blob input from the camera or other input device, and relaying this information through pre-defined protocols to frameworks which allow this raw blob data to be assembled into gestures that high-level language can then use to interact with an application. TUIO (Tangible User Interface Protocol) has become the industry standard for tracking blob data, and the following chapters discuss both aspects of multi-touch software: touch tracking, as well as the applications operating off of the tracking frameworks. B. Tracking Object tracking has been a fundamental research element in the field of Computer Vision. The task of tracking consists of reliably being able to re-identify a given object for a series of video frames containing that object (estimating the states of physical objects in time from a series of unreliable observations). In general this is a very difficult problem since the object needs to first be detected (quite often in clutter, with occlusion, or under varying lighting conditions) in all the frames and then the data must be associated somehow between frames in order to identify a recognized object. 1. Tracking for Multi-Touch Tracking is very important to multitouch technology. It is what enables multiple fingers to perform various actions without interrupting each other. We are also able to identify gestures because the trajectory of each finger can be traced over time, impossible without tracking. Thankfully, todays multitouch hardware greatly simplifies the task of tracking an object, so even the simplest filter in actuality becomes unnecessary. In fact much of the performance bottleneck of tracking systems tends to come from generating and maintaining a model of the background. Computational costs of these systems heavily constrict CPU usage unless 31

clever tricks are employed. However, with the current infrared (IR) approaches in multi-touch hardware (e.g., FTIR or DI), an adaptive background model turns out to be overkill. Due to the fact that the captured images filter out (non-infrared) light, much of the background is removed by the hardware. Given these IR images, it is often sufficient to simply capture a single static background image to remove nearly all of the ambient light. This background image is then subtracted from all subsequent frames, the resultant frames have a threshold applied to them, and we are left with images containing blobs of foreground objects (fingers or surface widgets we wish to track). Unlike general tracking solutions, we know that from one frame of video to another (~33ms for standard refresh rate of 30Hz), a human finger will travel only a limited distance. C. Gesture Recognition The future of Human-ComputerInteraction is the Natural User Interface which is now blurring its boundary with the present. With the advancement in the development of cheap yet reliable multi-touch hardware, it wouldnt be long when we can see multi-touch screens not only in highly erudite labs but also in study rooms to drawing rooms and maybe kitchens too. The mouse and the GUI interface has been one of the main reasons for the huge penetration of computers in the society. However the interaction technique is indirect and recognition based. The Natural User Interface with multi-touch screens is intuitive, contextual and evocative. The shift from GUI to Gesture based interface will further make computers an integral but unobtrusive part of our lifestyle. In its broadest sense, the notion of gesture is to embrace all kinds of instances where an individual engages in movements whose communicative intent is paramount, manifest, and openly acknowledged [17] Communication through gestures has been one of the oldest form of interaction in human civilization owing to various psychological reasons which, however, is beyond the scope of present discussion. The GUI systems leverages previous experience and familiarity with the application, whereas the NUI interface leverages the human assumptions and its logical conclusions to present an intuitive and contextual interface based on gestures. Thus a gesture based interface is a perfect candidate for social and collaborative tasks as

Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

well for applications involving an artistic touch. The interface is physical, more visible and with direct manipulation. However, only preliminary gestures are being used today in stand-alone applications on the multi-touch hardware which gives a lot of scope for its critics. Multi-touch interface requires a new approach rather than re-implementing the GUI and WIMP methods with it. The form of the gestures determines whether the type of interaction is actually multi-touch or singletouch multi-user. We will discuss the kind of new gesture widgets required, development of gesture recognition modules and the supporting framework to fully leverage the utility of multi-touch hardware and develop customizable, easy to use complex multi-touch applications. 1. Gesture Widgets Multi-touch based NUI setups provide a strong motivation and platform for a gestural interface as it is object based ( as opposed to WIMP) and hence remove the abstractness between the real world and the application. The goal of the interface should be to realize a direct manipulation, higher immersion interface but with tolerance to the lower accuracy implied with such an interface. The popular gestures for scaling, rotating and translating images with two fingers, commonly referred as manipulation gestures, are good examples of natural gestures [Fig.6.1.] [Fig.6.2.]. New types of gesture-widgets [Fig.6.3.][Fig.6.4.] are required to be build to fully implement the concept of direct manipulation with the objects (everything is an object in NUI with which the user can interact) and a evocative and contextual environment and not just trying to emulate mouse-clicks with a gesture.

Fig.6.2. Gesture Widgets Example [16] Gesture widgets should be designed with creative thinking, proper user feedback keeping in mind the context of the application and the underlying environment. These gesture widgets can then be extended by the application developer to design complex applications. They should also support customizable user defined gestures.

Fig.6.3. New Generation Gesture Widgets [16]

Fig.6.4. A View of New Gesture Widgets [16] 2. Gesture Recognition Process The primary goal of gesture recognition research is to create a system which can identify specific human gestures and use them to convey information or for device control. In order to ensure accurate gesture recognition and an intuitive interface a number of constraints are applied to the model.

Fig.6.1. Gestures [16]


Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

The multi-touch setup should provide the capability for more than one user to interact with it working on independent (but maybe collaborative) applications and multiple such setups working in tandem.

Fig6.5. Some Gesture Recognized by Touch screen [16] Gestures are defined by the starting point within the boundary of one context, end point and the dynamic motion between the start and end points. With multi-touch input it should also be able to recognize meaning of combination of gestures separated in space or time. The gesture recognition procedure can be categorized in three sequential processes: Detection of Intention: Gestures should only be interpreted when they are made within the application window. Gesture Segmentation: The same set of gestures in the same application can map to several meanings depending on the context of the touch events. Thus the touch events should be again patterned into parts depending on the object of intention. These patterned data will be sent to the gesture recognition module. Gesture Classification: The gesture recognition module will work upon the patterned data to map it to the correct command. There are various techniques for gesture recognition which can be used alone or in combination like Hidden Markov Models, Artificial Neural Networks, and Finite State Machine etc. D. Development Frameworks A number of frameworks have been released and are being developed to help in the development of multi-touch applications providing an interface for the management of touch events in an object-oriented fashion. However the level of abstraction is still till the

device input management, in the form of touch events being sent through TUIO protocol irrespective of the underlying hardware. The gesture recognition task which will realize the true potential of multi-touch surfaces, is still the job of the client. Often, however, some basic gestures are already included, in particular those for natural manipulation (e.g. of photos), but in general these frameworks arent focused on gestural interfaces. They rather tend to port the GUI and WIMP canons to a multitouch environment. Gesture and gesture recognition modules have currently gained a lot of momentum with the coming up of the NUI interface. Some of the important frameworks are: 1.Sparsh-UI [18] Sparsh-UI, published under LGPL license, seems to be the first actual multitouch gesture recognition framework. It can be connected to a variety of hardware devices and supports different operating systems, programming languages and UI frameworks. Touch messages are retrieved from the connected devices and then processed for gesture recognition. Every visual component of the client interface can be associated to a specific set of gestures that will be attempted to be recognized. New gestures and recognition algorithms can be added to the default set included in the package. 2. Grafiti Grafiti is a C# framework built on top of the Tuio client that manages multi-touch interactions in table-top interfaces. The possible use of tangible objects is particularly contemplated. It is designed to support the use of third party modules for (specialized) gesture recognition algorithms. However a set of modules for the recognition of some basic gestures is included in this project. 3. NUIFrame NUIFrame is a C++ framework based on the above discussed model (currently under development). It provides a separate gesture recognition module to which besides the touch-event sequence, contextual information regarding the view of the interface is also provided by the client application. This ensures that the same gesture on different objects can result in different operations depending on the context. It will also support custom gestures based on user specification for a particular command. The set of gesture widgets will also support automatic debugging 33

Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

by using pattern generation, according to the gestures supported by it. 4. AME Patterns Library The AME Patterns library is a new C+ + pattern recognition library, currently focused on real-time gesture recognition. It uses concept-based programming to express gesture recognition algorithms in a generic fashion. The library has recently been released under the GNU General Public License as a part of AMELiA (the Arts, Media and Engineering Library Assortment), an open source library collection. It implements both a traditional hidden Markov model for gesture recognition, as well as some reduced-parameter models that provide reduced training requirements (only 12 examples) and improved run-time performance while maintaining good recognition results. 5. Python[19] Python is a dynamic object-oriented programming language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools, comes with extensive standard libraries, and can be learned in a few days. Many Python programmers report substantial productivity gains and feel the language encourages the development of higher quality, more maintainable code. In many ways both Python and multi-touch have emerged for similar reasons of making computers easier to use. Although as a programming language Python certainly requires advanced technical knowledge of computers, a major goal is to let developers write code faster and with less hassle. Pythons dynamic nature, rapid development capabilities, make it an ideal language for prototyping and developing multi-touch interactions or applications quickly. VII. Multi-touch Applictions

A. Notebooks and tablets[1] 1. HP Touchsmart TX2 Released in December 2008, the Touchsmart tx2 was touted as the first consumer notebook and tablet PC with onscreen multi-touch control. 2. Dell Latitude XT In July 2008, Dell released multitouch touch-screen drivers for the Latitude XT Tablet, claiming the "industrys first convertible tablet with multi-touch 34

capabilities [20]. Dell has partnered with multi-touch technology startup N-trig to enable multi-touch capabilities for its tablet. There have been numerous reports of serious problems with the XT multi-touch drivers or hardware interface, which, as of October 2008, remain unresolved by Dell. The major issue is cantered on the multi-touch drivers failing when the computer is booted when USB devices are connected at boot, or when certain internal media drives are present. Often, but not always, the issue is resolved if the computer is booted without the USB devices and they are connected after boot. A similar, but not identical, issue may also be caused by interfering software, such as the iTunes helped. Issue resolution is complicated by the possible failure of the driver install program to function in the presence of damaged drivers, requiring manual removal of related files and registry keys. See External Links for links to user discussions of this issue. With the introduction of Windows 7 which offers native multi-touch support, preliminary tests have shown that many of these issues with the XT have been solved, together with a much improved N-Trig Win 7 driver. 3. Sharp Mobius In April 2009, Sharp unveiled the first laptop with an optical-sensor LCD pad. The optical-sensor LCD technology combines LCD and CCD elements within its pixels and can detect fingers and objects such as a stylus, and is claimed to be able to scan a business card placed on top of the screen. Further improvements to this function are expected to enable fingerprint authentication in the future. 4. Lenovo Tablets (x200) Currently, Lenovo's definition of "Multitouch" implies that the screen responds to either a finger or a digitizer pen. However, it is not capable of detecting simultaneous touch points on the tablet screen nor any forms of multi-finger gestures, while the Windows 7 already supports simple dual-touch ones. B. Software[1] 1. Mac OS X Tiger Apple Mac OS X 10.4, as well as derivatives such as iPhone OS, support multitouch. This extends from the windowing environment, through to Safari, iLife and other software.

Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

2. Linux Various versions of Linux, and especially those versions used on small devices such as the Asus Eee PC and Palm Pre, have multi-touch support. 3. Windows 7 While previous versions of Windows do not support multi-touch screens, Windows 7 will support multi-touch when released [21]. The operating system is known to have a multi-touch mapping application, photo viewing program, and incorporation in Internet Explorer 8. In January 2009, Microsoft joined with other investors who invested twenty-four million dollars in N-Trig Ltd., which plans to make computer hardware that takes advantage of Windows 7's multi-touch support.

Fig.7.1. A multi-touch photos application is demonstrated in a pre-release version of Windows 7 [1] 4. 22MILES One of the advantages of 22MILES multi-touch framework is OS and multi-touch hardware independent. Their multi-touch software can run Windows 7, XP, Vista, Ubuntu Linux and Mac OS X, while it supports various multi-touch hardware technology, such as Infrared Matrix/Camera, Projected Capacitive, SAW (dual-touch), resistive, FTIR/ DI/ DSI, LED-LP. But unlike open-source, their proprietary software and multitouch gestures only target to business, such as digital signage, Medical Industry, GIS, Real Estate, and so on. 5. Multitouch for Java (MT4j) Another multi-touch framework is MT4j. It is cross-plattform (XP, Vista, and Linux) and supports various multi-touch hardware technology based on the TUIO protocol. MT4j is open source. MT4j supports application development in 2D and 3D and many common input formats such as SVG and 3ds. Using OpenGL for most rendering work, MT4j based applications perform very well in combination with a hardware-accelerated graphics card.

C. Phones and Music players: 1. Apple iPhone The iPhone is an Internet-connected, multimedia GSM smartphone designed and marketed by Apple Inc. Because its minimal hardware interface lacks a physical keyboard, the multi-touch screen renders a virtual keyboard when necessary. The iPhone functions as a camera phone (also including text messaging and visual voicemail), a portable media player (equivalent to a video iPod), and an Internet client (with email, web browsing, and Wi-Fi connectivity). The firstgeneration phone hardware was quad-band GSM with EDGE; the second generation added UMTS with 3.6 Mbps HSDPA; the third generation adds support for 7.2 Mbps HSDPA downloading but remains limited to 384 Kbps uploading as Apple had not implemented the HSPA protocol. 2. Apple iPod Touch[22] The iPod Touch (trademarked and marketed as iPod touch) is a portable media player, personal digital assistant, and Wi-Fi mobile platform designed and marketed by Apple Inc. The product was launched on September 5, 2007 at an event called The Beat Goes On. The iPod Touch adds the multitouch graphical user interface to the iPod line. It is the first iPod with wireless access to the iTunes Store, and also has access to Apple's App Store, enabling content to be purchased and downloaded directly on the device. 3. Palm Pre[23] The Palm Pre is a multimedia smartphone designed and marketed by Palm with a multi-touch screen and a sliding keyboard. The phone was launched on June 6, 2009, and is the first to use Palm's new Linuxbased[1] operating system, webOS. The Pre functions as a camera phone, a portable media player, a GPS navigator, and an Internet client (with text messaging, email, web browsing, and local Wi-Fi connectivity). 4. Palm Pixi[24] The Palm Pixi is a multimedia smartphone under development by smartphone maker Palm. The device is viewed as a successor to the Palm Centro smartphone and will be Palm's second webOS device, after the Palm Pre. The device was officially announced on September 8, 2009, and is expected to be released by the end of 2009.


Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

5. Zune HD[25] The Zune HD is a Portable Media Player in the Zune product family by Microsoft released on September 15, 2009 in 16GB and 32GB capacities.[4] It utilizes a touchscreen interface for navigation and includes Wi-Fi for synchronization, access to the Zune Marketplace and Web browsing. The Zune HD utilizes the Nvidia Tegra APX 2600 chip, allowing it to play 720p video through the optional HDMI Zune dock on a highdefinition television. Otherwise, content will be scaled down to 480x272 pixels on the player's OLED screen. VIII. FUTURE SCOPE OF MULTITOUCH A. Classrooms of the future to have multitouch desks

Fig.8.1. Future Classrooms Researchers at Durham University in the UK are working hard to ensure that the academic sector isn't left out of the multi-touch craze. The still-unnamed "interactive multitouch desk" is a major part of the SynergyNet framework, an interactive classroom environment they're building based on a gaming engine called jMonkey. There's no word on how soon SynergyNet will change the world, but when it does the software will be open source -- allowing anyone to get into the code and tweak it to their liking. In the meantime, schools looking for a "high tech initiative" of their own will have to be content with throwing laptops at their students. B. Apples vision of future multi-touch[26] Apples vision of future multi-touch includes voice, sight, and finger-recognition. Leave it to the pioneers of multi-touch in the mobile space to apply for a patent on all future versions of multi-touch. Apple (NSDQ: AAPL)s latest patent filing suggests that Cupertino is working to integrate more than just touch-based input modalities in to future versions of multi-touch. The problem with the current multi-touch setup is that its tailored to

touch-based mechanics alone. Multi-touch on the iPhone and iPhone 3G is goo for mechanical manipulation, but without complementary input modalities like voice or finger-recognition, the current multi-touch setup falls short of offering truly immersive user interaction. Imagine trying to resize and rotate a picture while changing its color and inserting a bit of text. It wouldnt make sense to resize and rotate the picture using your voice. There are size and rotation-degree parameters that are just more well-suited for manipulation through touch-based multi-touch technology its just easier to pinch/stretch and rotate the picture with your finger than with voice commands. Likewise, its easier to use voice commands to change the colour of the background picture or add a text string to the picture. Apple proposes, in their latest patent application, that future multi-touch technologies will integrate almost all foreseeable input modalities to complement touch-based multi-touch. Mechanical manipulations could be assigned to touchbased commands, while voice commands are relegated to functions that require choosing an option from a list. Fig . 8.2 .

Image Resizing [26]

Fig.8.3. Using Voice command and MultiTouch Together [26]


Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

Furthermore, Apple envisions future multi-touch setups incorporating fingerrecognition that would allow the user to assign different functions to each finger which would allow for more complex commands to be issued. Gaze-vector tracking could allow your future laptop or mobile phone to recognize which menu youd like to select and automatically drill down to the appropriate category. Then theres the proposition of using finger-tracking and force-sensing technologies that would allow the user to affect different degrees of control depending on fingerpressure and position. Combine all that with accelerometer support and some pixie dust and youve got the makings of a seriously immersed control environment. We cant wait for a future filled with multi multi-touch and pixie dust! IX. REFERENCES


[10] [11]

[12] [13] [14] [15] [16] [17] [18] [19] [20]

[1] (accessed on 12 October 2009) [2] Buxton, Bill. 2008. Multi-Touch Systems that I Have Known and Loved. rview.html (accessed on 29 September 2009). [3] (accessed on 12 October 2009) [4] "Multi-touch comes to resistive touchscreens", i-touch-comes-to-resistive-touchscreens (accessed on 29 September 2009) [5] Patschon, Mark (1988-03-15), Acoustic touch technology adds a new input dimension, Computer Design(accessed on 12 October 2009) [6] Please Touch! Explore The Evolving World Of Touchscreen Technology". cfm?AD=1&ArticleID=18592. (accessed on 12 October 2009) [7] Kable, Robert G. (1986-07-15), Electrographic Apparatus, United States Patent 4,600,807, (accessed on 12 October 2009) [8] Minsky,M.R. (1984-07-01). "Manipulating simulated objects with real-world gestures


[22] [23] [24] [25] [26]

using a force and position sensitive screen". Computer Graphics. (accessed on 12 October 2009) Keuling, Christopher (2008-11-03). "Touchscreens Press Deep Into Consumer Electronics". ECN Magazine. (accessed on 12 October 2009) Sensors help make ticket machines vandal proof". Engineeringtalk (accessed on 12 October 2009) Beyers, Tim (2008-02-13). "Innovation Series: Touchscreen Technology". The Motley Fool. (accessed on 12 October 2009) Acoustic Pulse Recognition Touchscreens, Elo Touch Systems, 1888-07-31 (accessed on 12 October 2009) (accessed on 12 October 2009) rface (accessed on 12 October 2009) (accessed on 12 October 2009) (accessed on 12 October 2009) flosc: Flash Open Sound Control (accessed on 12 October 2009) (accessed on 12 October 2009) (accessed on 12 October 2009) Dell Propels Laptop Innovation With First Multi-Touch Convertible Tablet, Larger Solid State Drive (accessed on 12 October 2009) Windows Vista Team Blog. "Microsoft demonstrates Multi-touch". MSDN Blogs. Retrieved 2008-05-28. (accessed on 12 October 2009) (accessed on 12 October 2009) (accessed on 12 October 2009) (accessed on 12 October 2009) (accessed on 12 October 2009) ples-vision-of-future-multi-touchincludes-voice-sight-fingerrecognition.html (accessed on 12 October 2009)


Journal of Sci. Engg. & Tech. Mgt. Vol 2 (2), July 2010

[27] Nakatani, L. H., John A Rohrlich (1983). "Soft Machines: A Philosophy of UserComputer Interface Design". Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI83): 12-15. [28] Jeff Han, Multi Touch interaction Research (accessed on 13 october 2009) X. Author Biographies Dr. H. B. Kekre has received B.E. (Hons.) in Telecomm. Engineering. from Jabalpur University in 1958, M.Tech (Industrial Electronics) from IIT Bombay in 1960, M.S.Engg. (Electrical Engg.) from University of Ottawa in 1965 and Ph.D. (System Identification) from IIT Bombay in 1970 He has worked as Faculty of Electrical Engg. and then HOD Computer Science and Engg. at IIT Bombay. For 13 years he was working as a professor and head in the Department of Computer Engg. at Thadomal Shahani Engineering. College, Mumbai. Now he is Senior Professor at MPSTME, SVKMs NMIMS University. He has guided 17 Ph.Ds, more than 100 M.E./M.Tech and several B.E./ B.Tech projects. His areas of interest are Digital Signal processing, Image Processing and Computer Networking. He has more than 250 papers in National / International Conferences and Journals to his credit. Presently He is Fellow of IETE and Life Member of ISTE Recently six students working under his guidance have received best paper awards. Currently ten research scholars are pursuing Ph.D. program under his guidance. Sudeep D. Thepade has Received B.E.(Computer) degree from North Maharashtra University with Distinction in 2003. M.E. in Computer Engineering from University of Mumbai in 2008 with Distinction, currently Perusing Ph.D. from SVKMs NMIMS University, Mumbai. He 38

has more than 06 years of experience in teaching and industry. He was Lecturer in Dept. of Information Technology at Thadomal Shahani Engineering College, Bandra(w), Mumbai for nearly 04 years. Currently working as Assistant Professor in Computer Engineering at Mukesh Patel School of Technology Management and Engineering, SVKMs NMIMS University, Vile Parle(w), Mumbai, INDIA. He is member of International Association of Engineers (IAENG) and International Association of Computer Science and Information Technology (IACSIT), Singapore. His areas of interest are Image Processing and Computer Networks. He has about 45 papers in National/International Conferences/Journals to his credit with a Best Paper Award at International Conference SSPCCIN-2008 and Second Best Paper Award at ThinkQuest-2009 National Level paper presentation competition for faculty. Digvijay Singh is pursuing B.Tech. (Computer Science) degree from Mukesh Patel School of Technology Management and Engineering, SVKMs NMIMS University, Mumbai. His interest in Multi Touch Technology developed during the seminar report generation as part of third year B.Tech curricula. Tushant Mitaal is pursuing B.Tech. (Computer Science) degree from Mukesh Patel School of Technology Management and Engineering, SVKMs NMIMS University, Mumbai. His interest in Multi Touch Technology developed during the seminar report generation as part of third year B.Tech curricula. Aman Pratap Singh is pursuing B.Tech. (Computer Science) degree from Mukesh Patel School of Technology Management and Engineering, SVKMs NMIMS University, Mumbai. His interest in Multi Touch Technology developed during the seminar report generation as part of third year B.Tech curricula.