2012 International Conference on Systems and Informatics (ICSAI 2012)
A Multi-Touch Natural User Interface Framework
Jialiang Yao Terrence Fernando
Thinklab Thinklab
University of Salford University of Salford
Salford, UK Salford, UK
Abstract—This paper presents the design and implementation of
a multi-touch gesture interface framework that has been
designed and implemented for collaborative urban planning. The
user gestures required for supporting collaborative urban
planning have been defined by analysing a well established urban
planning environment, previously developed by the authors.
Although, the urban planning have been considered as the
application context, the overall multi-touch gesture interface
framework has been designed and implemented as a layered
architecture to support any high level applications. This layered
gesture architecture comprises a multi-touch raw data layer, a
basic gesture layer and an application specific gesture layer.
Keywords: Multi Touch; NUI; gesture design; gesture
framework
I. INTRODUCTION
The term natural user interface (NUI) is used to refer to
interfaces which allow the user to interact with a system based
on the knowledge learnt from using other systems [1].
Typically, NUI can be aided by technologies allowing users to
carry out natural motions, movements or gestures to control the
computer application or manipulate on-screen content. A multi-
touch device is one of the technologies that have emerged
recently and is widely used in creating NUI for developing
interactive computer applications.
Multi-touch devices enable users to directly interact with
information on screens using fingers as input devices. It
provides users with a stronger feeling of control over their
interactions rather than being controlled by the system [2].
Multi-touch interaction is becoming common as a type of user
interface for many software applications running on mobile
devices such as tablets and smart phones. Furthermore, due to
its popularity and ease of use, the multi-touch input method
opens up possibilities for a broader range of applications where
users need to participate and collaborate together to explore
alternative solutions and build consensus.
This research explores how a multi-touch interface
framework can be developed to support intuitive user
interaction during collaborative urban planning discussions.
Since many stakeholders, with varying levels of computing
knowledge are involved in collaborative urban regeneration
projects, this research explores the development of a multi-
touch interface that is based on natural gestures, requiring little
training for exploring the urban spaces and proposed designs.
This research builds on the authors’ previous research on
the COllaborative Planning Environment (COPE) which
Hongxia Wang
School of Built Environment
University of Salford
Salford, UK
exploited the power of PowerWall technology [3] for
stakeholder engagement in urban planning. The interaction and
team collaboration within the COPE is currently supported
through a gamepad interface. This paper presents the authors’
approach for enhancing the current user interaction capability
by exploiting the power of natural hand gestures. Furthermore,
it presents a generic multi-touch based natural user interface
framework that provides a range of high level gestures that can
be used to develop interactive information exploration in
applications such as urban planning.
II. REVIEW
During recent years, there have been many research and
development efforts towards creating multi-touch natural user
interfaces to enhance user interaction and collaboration.
However, designing and implementing a multi-touch natural
user interface is still a challenging task. In this research the
authors’ main concerns are the selection of a set of natural
gestures appropriate for urban planning tasks and the
development of a multi-touch gesture framework that can
easily be deployed in supporting geo-spatial applications such
as urban planning.
The design of multi-touch gesture sets have been
investigated by some researchers, such as Ringel et al. and Wu
et al [4, 5]. Hinrichs et al. observed many different gesture
instances from a field study and categorised them into a group
of low-level actions: drag/move, enlarge/shrink, rotate, tap,
sweep, flick, and hold [6]. Existing multi-touch based systems,
such as Apple iOS and Android systems, also impact on the
gesture set design. Due to their increasing popularity, certain
gestures have become widely known by the general public.
Other related work, such as GestureWorks [7], has also
influenced gesture sets and their possible meanings. Although
the selection of a gesture set mainly depends on the application
requirements, all of these efforts have played a significant role
in influencing the nature of multi-touch based applications and
the design features of multi-touch based user interfaces.
Some multi-touch gesture frameworks have been developed
to provide a development platform for various applications.
Three of the most popular multi-touch gesture frameworks are:
an open source framework – Tangible User Interface Objects
(TUIO), Windows 7 SDK from Microsoft [8], and PQ Labs
multi-touch SDK (http://multi-touch-screen.com/sdk.html).
TUIO is an open framework that defines a common protocol
and API for tangible multi-touch surfaces. The TUIO protocol
allows the transmission of an abstract description of interactive
499 978-1-4673-0199-2/12/$31.00 ©2012 IEEE
surfaces, including touch events and tangible object states.
However, TUIO does not provide gesture analysis. As a result
it has to be implemented by users. There is a wide range of
applications which have used TUIO listed on the TUIO web
site (http://www.tuio.org/). To the best of the authors’
knowledge, no suitable TUIO-based framework can provide
the desired support to the applications within the context of
collaborative urban planning. Both Windows 7 SDK and PQ
Labs provide either raw touch data support or very basic
gestures’ support which is very limited for complex
applications such as urban planning. Developers need to further
develop gesture operations for their own applications using
these SDKs.
III. NATURAL USER INTERFACE DESIGN FOR URBAN
PLANNING
The design and implementation of the natural interface is
being carried out as an iterative process and this paper presents
the outcome of the first cycle of the iterative process. Seven
subjects (three academics and four researchers), who closely
work with urban planning teams in using the COPE technology
platform, were used to analyse the typical interaction tasks
during urban planning and mapping them to natural gestures.
TABLE I. KEY UI FUNCTIONS USED IN COPE FOR URBAN PLANNING
Key Interaction
Purpose
functions Features
Exo-centric To allow users to fly around the Move: left, right,
navigation urban space to explore green up, down.
spaces, road layout and other Rotate : pitch, yaw
urban data sets such as crime, Zoom : in, out
unemployment and health
Ego-centric To allow users to walk along a Walk / drive along a
navigation street or a urban space to pointed direction.
understand the proposed spatial
changes to the space or to explore
social challenges in a given
neighbourhood.
Area To allow the users to specify an Ability to mark a
selection area of interest with a view to boundary
uploading physical data such as
3D buildings, and social data
such as crimes and
unemployment .
Select objects To allow users to interrogate the Mouse
data attached to a building (such virtual pointer
as address, energy, ownership) or
interrogate detailed data behind
visual icons.
Select menu To allow users to select and Mouse click, virtual
widgets populate the urban space with a pointer
particular type of social data from
a menu
Remove data To allow users to delete uploaded Menu option
data (buildings or social data) to remove data
bring new data sets
A. Functional Analysis of the COPE’s Environment
At present, the COPE urban planning environment allows
users to load a particular urban space, created by combining
terrain data, aerial photographs, building models and social
data sets, and to explore the space during urban planning
discussion meetings. The key functions that are supported by
the COPE environment can be summarised as:
• Navigation (with both exo-centric and ego-centric
mode) – this allows the user to navigate
• Object and menu selection/query
• Area selection
Table I below shows the task analysis conducted by
researchers to identify the key interaction techniques used
during a typical urban analysis session.
At present, the above actions are supported through a
gamepad and therefore the challenge was to identify a natural
interface based around the multi-touch display. The next
section explains the authors’ approach for prototyping a
gesture-based interface to support the above functions.
B. Defining Gestures
In order to identify natural gestures for developing the first
prototype, a workshop was organised involving subjects who
are usually involved in driving the COPE environment or
working with urban planning teams.
Prior to the workshop, the authors completed a literature
review of gesture applications. This literature review searched
the common gestures discussed and designed by other research
projects (such as GestureWorks.com) as well as some basic
gesture design guidelines which emphasised key aspects such
as familiarity, searchability, expressivity [9] and simplicity
[6]. Important suggestions by Hinrichs [6] that gesture types
should be classified based on the relation of the touching
fingers and their movement instead of the number of fingers,
were identified. Real world examples researched by the authors
included applications on the Microsoft surface table,
demonstration programmes from PQLabs as well as
applications on the Apple iOS.
During the workshop, the literature review results, the
gesture design guidelines, and real world examples as well as
the outcome of the functional analysis of the COPE
environment were presented and discussed by the participants.
Since the idea of this research was to develop a natural
interface that is familiar to users, it was decided to use the user
click,
interaction with a paper map on a fixed table as the basis for
developing a gesture interface for the urban planning
environment. This physical interaction metaphor was
considered as being the closest to the manipulation of urban
maps on a multi-touch table. The participants were asked to
consider the following questions:
• How would you perform the interaction operations
described in Table I on a physical map?
to
• If the paper is an e-paper, what additional gestures
would you use to interact with the map?
Table II below summarises the proposals that were derived
by the participants in mapping the key interaction operations
into gestures.
The above user feedback was used as the basis for creating
a natural gesture interface to support the interaction with the
500
urban planning environment. The following section explains
the final gestures designed for the application.

TABLE II. GESTURE RECOMMENDATION

Recommended natural gestures (based on paper (a) (b) (c)
Interaction Task
metaphor)
Move: left, right, Move the paper by using one or more fingers
up, down
Rotate: about the Rotate the paper by touching it and then rotating by
axis perpendicular two or more fingers
to the paper.
Rotate: about a Pin down the paper by one or more fingers from one (d) (e) (f) (g)
given point hand and then rotate the paper by using the fingers Figure 1. Examples of navigation gestures. (a) and (b) move gestures, (c)
(one or more) from the other hand and (d) zoom gestures, (e) – (g) rotate gestures
Rotate (tilt): tilt Pin down one point using a finger from one hand and
the paper about a tilt the plane by applying a directional movement
point from the fingers from the other hand
Mark a boundary Pin down the paper by one or two fingers from one
hand and use one finger from another as a pen to
draw the boundary
Delete data Use palm with finger together as a eraser to delete
data
Walk / drive along Move a finger along the path that the user wants to
a pointed travel Figure 2. Tilt gesture
direction.
Select object or Touch the object using any finger However, all the above navigation operations are not
menu widget
mutually exclusive. Sometimes, compound actions such as
Zoom (in and out) Use the pinch motion and its reverse to zoom in and
out. This gesture was heavily influenced by similar move-and-zoom, move-and-rotate can happen at the same time
gestures used in many mobile phones which is showed in Fig. 3. In which case, all actions will be
performed in this design.
C. Gesture design
Gestures identified in the previous section were analysed
and designed to support the functions presented in Table I,
ensuring each gesture is mapped to only one type of operation.

1) Navigation
a) Exo-centric navigation
In the exo-centric navigation mode, typical navigation
functions include move, zoom and rotate (Table I). The
gestures for these operations are defined as moving fingers in
parallel, towards or apart from each other or rotating around a
point respectively. In this design, the number of fingers to be
used are not defined which means the user can use any number
of fingers to do such operations. However, the zoom and rotate
operations cannot be operated with only one finger; they
require a minimum of two fingers (Fig. 1) 1.
Although, the forwards and backwards tilting of the map is
a rotational operation, a different gesture is required to allow
the user to rotate the normal of the map away from the screen.
For this special action, when the user pins down the map using
one finger, a vertical scroll bar will be prompted on the screen.
Moving another finger within this bar will tilt the map (Fig. 2).
Figure 3. Example of move-and-zoom gesture happening at the same time
b) Ego-centric navigation
In ego-centric navigation mode, only “driving/walking”
functions are required. This function is mainly used to simulate
one who walks (moves and rotates) along a street. In this mode,
the gestures are designed to use single touch to do both the
navigation and the rotation. When a finger is placed on the
screen, the first touch point is used as the reference point of
navigation. Then the finger will move around the reference
point to control the navigation (Fig. 4). The distance between
the current touch point and the reference point indicates the
moving velocity. The angle between the “up vector” and the
vector along reference point and the touch point defines the
rotation factor.

1
Some of the gesture figures are taken from GestureWorks [7]. When a full
hand is used in a gesture in these figures, it indicates that any number of
fingers can be used. Otherwise, it indicates the exact number of fingers used to Figure 4. Ego-centric navigation control
express a gesture

501
2) Undo or erase
This gesture imitates the action of erasing drawings from a
piece of paper. Using a palm to touch the screen, and moving
the palm left and right will carry out the undo or erase
operation (Fig. 5). This operation will undo one action. To
undo again, the palm must be lifted off the screen first before
the gesture is repeated.
Figure 5. Undo or erase gesture.
3) Area selection
Figure 6. Line-drawing gesture
Like a pin attaching paper to a board, the paper can rotate
about the pin but cannot move. However, if there are more than
two pins used to attach the paper to the board, the paper will be
fixed on the board. Based on this idea, the gesture for area
selection also comes from the way of handling paper on a
board. Users can use two or more fingers to hold the map, and
then use another finger moving on the screen to draw the
boundary of an area (Fig. 6). The end of the gesture is defined
as the time when the user’s hand is removed from the screen.
However, if the user changes his/her mind, he/she can use the
undo/erase gesture to cancel the area selection operation before
releasing the ‘holding gesture’ from the screen.
4) Object selection / query
(One-finger) single-tap and (one-finger) double-tap gestures
are used to carry out object selection and query tasks. The
experience of tap and double tap come from using a mouse’s
click and double-click action in a GUI (Graphical User
Interface). A single click usually selects (or highlights) an
object while double-click usually executes or opens the object
in many commonly used graphical user interfaces, such as the
GUI in Windows and Mac OS.
Similar to the mouse action of ‘click’ and ‘double click’,
single-tap and double-tap gestures were designed with the
following in mind:
• Single-tap is the action of placing a finger on the
screen and then its removal within a certain time
interval, similar to the action of a mouse ‘click’.
• Double-tap is the action of placing a finger on the
screen and removing twice within certain spatial and
temporal intervals, which is similar to the action of
mouse ‘double-click’.
502
By using these four types of gestures, the multi-touch
device can control the digital map like a floating piece of paper
map being handled with two hands.
IV. THREE-LAYER FRAMEWORK FOR MULTI-TOUCH UI
A. Framework design
One of the objectives of this work is to explore a general
purpose multi-touch framework to support NUI for different
applications. Rather than developing all possible gestures, this
framework provides the possibility of aggregating low level
gestures to appropriate high level gestures, depending on the
application and user requirements. The authors’ solution is to
break gestures into limited numbers of low level gestures first.
Then higher-level gestures are recognised by combining a
sequence of low level gestures.
In this work, the application level gestures are broken into
several gesture “atoms”. The gesture atoms are basic “pure”
gestures. A series of atom gestures can be grouped together to
form an application gesture to represent a certain functional
meaning.
The basic gesture information is classified into two
categories: dynamic gestures and touch information. Dynamic
gestures are formed by moving touches, such as finger-down,
finger-up, move, zoom, rotate, etc. A touch information set
provides all the touch points’ information (e.g. position,
moving/non-moving, moved before, touch area, etc.) The two
types of information are used to define application gestures.
Therefore, the multi-touch framework is organised into
three layers which is illustrated in Fig. 7. The left part in Fig. 7
shows the structure of the framework. The right part of Fig. 7
shows its relation to a COPE based application on a Windows
platform.
Multi-Touch Frame Work An OSG Application
Application Gesture Layer OSG Event Queue
Basic Gesture Layer
Windows Event Queue
Multi-Touch RAW Layer
Multi-touch Events Other events
Events
Figure 7. The structure of the multi-touch framework
• The 1st layer (the lowest layer) is the RAW layer,
which represents the RAW information from the
multi-touch device. It is usually the interface that the
device driver provides, such as PQLabs and TUIO
(3rd party library).
• The 2nd layer is the basic gesture recognition layer,
which provides basic gesture atoms/elements together
with touch information for the upper layer.
 • The 3rd layer is the application layer. This layer
provides the final user interface based on the gestures
captured by the 2nd layer. It uses the gestures to
interact with the context of the application and
provides the special user interface functions required
by the application. Therefore, the 3rd layer is an
application related layer, and cannot be typically used
for general purposes.
B. Gestures’ definition within each layer
1) RAW layer
The RAW layer gestures are usually the initial touch
information, such as the touch id, orientation, touch size and
shape, touch image for each touch on the screen surface. The
details of the data are usually decided by the touch screen’s
capability and provided to the application over the device
driver or 3rd party libraries.
Because of the data structures of different libraries, the data
may need to be restructured for them to fit into the higher
layers.
2) Basic gesture layer
The basic gesture layer tries to capture gestures from raw
touch information and generate basic gesture information for
the application layer.
This layer first generates statistical information from the
raw touch information, such as the number of moving touches,
the number of non-moving touches and each touch’s history.
Then it captures the basic gesture based only on moving
touches.
The gestures that the 2nd layer provides include:
• Parallel move
• Split
• Rotate
• Finger-down, finger-up, single tap and double tap for
each touch
• Begin, end and inertia begin
Because most of the current multi-touch devices cannot
distinguish touches from different people, it assumes that all
touches come from the same person. In general, the framework
should also provide touch-grouping functions to support
collaboration.
3) Application layer
The 3rd layer provides functions to drive the multi-touch
user interface as mentioned in section 3. It receives basic
gestures from the 2nd layer, and combines them with the
context underlying the touches to generate proper actions for
the application. The gesture set defined in section 3 is
recognised in this layer and the intended user action is passed
to the application to be executed, and then executed in response
to the gesture.
V. INITIAL NUI PROTOTYPE IMPLEMENTATION
To test the feasibility of the multi-touch NUI framework,
the authors have implemented and integrated the framework
with the COPE environment.
The COPE implementation is based on Open Scene Graph
(2.8) on a Microsoft Windows platform. Since the OSG does
not process multi-touch events, one of the key tasks of the
prototype is to act as a plug-in for the OSG-based COPE
architecture so that the multi-touch events are accepted without
modifying any code of the OSG itself.
In order to implement the NUI plug-in for the COPE, the
multi-touch events need to be captured. At same time, the NUI
prototype also needs to distinguish the multi-touch events from
any other types of events, such as mouse events, so that the
original interface still works.
The implementation approach is based on two steps. The
first step is to capture multi-touch events and place them into
the OSG’s event queue. The events cannot be placed into the
Windows’ event queue because the OSG can only put very
limited types of Windows’ events into the OSG event queue,
and all other types of events are thrown away. The second step
is to process those multi-touch events in an extended
manipulator which is based on the OSG’s event manipulator. In
the implementation of the COPE prototype, a new manipulator,
derived from the OSG’s matrix-manipulator, has been
implemented so that a specially designed user interface could
be implemented. It supports a plug-in structure by
implementing an event manipulating chain which dispatches
events first to the plug-ins’ manipulator. If not processed, it
then dispatches them to the original event manipulator.
Figure 8. The COPE multi-touch user interface
The results show that the plug-in type implementation can
add a new interface into a software system without affecting
any existing user interface. The physical look and feel of the
interface is shown in Fig. 8.
VI. DISCUSSION AND FUTURE WORK
This paper presented the design of a gesture set for urban
planning applications and the development of a multi-touch
natural user interface framework that could support high level
application gestures. A prototype has been implemented to
503
transform the COPE urban planning environment into a multi-
touch supported application.
The three-layer framework provides a generic structure for
the development of a multi-touch user interface. The
implementation of the prototype based on the COPE has
demonstrated the capability of the framework.
The prototype has been informally tested by a small group
of end-users. During the tests, the authors have intentionally
asked users to use this system without offering prior knowledge
of the multi-touch gestures. The initial results show that there is
little difficulty for these users to navigate around (in the exo-
centric mode) with multiple fingers which includes move,
zoom and rotate. Object selection with the tap gesture was also
carried out without difficulty. However, the less commonly
used gestures, such as the hold gesture, the tilt operation, the
erase and area selection required prompting and training. Many
users were simply unaware of the presence of such functions.
After introducing these gestures the users could use them with
ease.
The next step of the authors’ research is to carry out a
formal usability evaluation and to integrate the framework with
other applications to validate the framework as a general
purpose system and enhance its usability.
REFERENCES
[1] D. A. Norman, “Natural user interfaces are not natural,” Interactions,
vol. 17, pp. 6-10, 2010.
504
[2] S. Bachl, M. Tomitsch, C. Wimmer, and T. Grechenig, “Challenges for
designing the user experience of multi-touch interfaces,” presented at the
Engineering Patterns for Multi-Touch Interfaces Workshop (MUTI'10)
of the ACM SIGCHI Symposium on Engineering Interactive Computing
Systems, Berlin, Germany, 2010.
[3] J. Yao, T. Fernando, H. Tawfik, R. Armitage, and I. Billing, “A VR-
centred workspace for supporting collaborative urban planning,” in 9th
International Conference on Computer Supported Cooperative Work in
Design, Coventry, UK, 2005.
[4] M. Ringel, K. Ryall, C. Shen, C. Forlines, and F. Vernier, “Release,
relocate, reorient, resize: fluid techniques for document sharing on multi-
user interactive tables,” presented at the CHI '04 extended abstracts on
Human Factors in Computing Systems, Vienna, Austria, 2004.
[5] M. Wu, C. Shen, K. Ryall, C. Forlines, and R. Balakrishnan, “Gesture
registration, relaxation, and reuse for multi-point direct-touch surfaces,”
presented at the Proceedings of the First IEEE International Workshop
on Horizontal Interactive Human-Computer Systems, 2006.
[6] U. Hinrichs and S. Carpendale, “Gestures in the wild: studying multi-
touch gesture sequences on interactive tabletop exhibits,” presented at
the Proceedings of the 2011 Annual Conference on Human factors in
Computing Systems, Vancouver, BC, Canada, 2011.
[7] GestureWorks - Multitouch Framework - Build Gesture-Driven Apps.
http://gestureworks.com.
[8] Y. Kiriaty, MultiTouch Capabilities in Windows 7, MSDN Magazine
http://msdn.microsoft.com/en-us/magazine/ee336016. aspx , 2009.
[9] A. Bragdon, R. Zeleznik, B. Williamson, T. Miller, and J. J. LaViola Jr.,
“GestureBar: Improving the Approachability of Gesture-based
Interfaces,” presented at the Proceedings of the 27th International
Conference on Human Factors in Computing Systems, Boston, MA,
USA, 2009.