Participatory Design of Touch Gestures for Informational
Search on a Tablet Device
Tsele Rakubutu Helene Gelderblom
University of the Witwatersrand University of Pretoria
Dept of Information Systems Department of Informatics
Johannesburg, South Africa Pretoria
27 (0)11 717-8164 27 (0)12 4203352
TseleRakubutu@gmail.com helene.gelderblom@up.ac.za
ABSTRACT
This study set out to answer the question: what would a gesture
set for conducting an informational search on a multi-touch tablet
web browser look like if designed in collaboration with users with
limited touch screen experience? In addressing this question, we
developed such a user-defined gesture set, based on gestures
elicited from participants with little or no experience with touch
screen devices. Focusing on 24 specific search-related browser
functions, 20 participants were asked, in a lab setting, to
experiment with and then suggest a gesture for each of the
functions. Data collection included video and audio recordings,
think-aloud data, informal interviews and self-evaluation of
proposed gestures. A combination of qualitative and quantitative
analysis revealed the following: one-handed gestures are preferred
over two-handed gestures; users who have mostly been exposed to
mouse interaction prefer gestures that correspond to point-and-
click actions; completion of a task should be possible with more
than one gesture; and complex tasks should allow varying
combinations of gestures).
Categories and Subject Descriptors
H.5.2 [Information interfaces and presentation]:
User interfaces – interaction styles, user-centered design.
General Terms
Design, Experimentation, Human Factors.
Keywords
Multi-touch, Tablet, Web browsing, Participatory design, Gesture
design, Referents.
1. INTRODUCTION
Many touch screen applications have their own specific gesture
sets that are used to interact with digital objects and provide input
[4]. Such a gesture set is used throughout the application or
interface and is also called a gesture vocabulary [8]. Many gesture
sets use varying combinations of one finger tap interactions,
which suggests that they have simply been adapted from the
pointer-based paradigm used in popular operating systems, such
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to
post on servers or to redistribute to lists, requires prior specific
permission and/or a fee. Request permissions from:
Permissions@acm.org.
SAICSIT2014, September 29 - October 01 2014, Centurion, South Africa
Copyright 2014 ACM 978-1-4503-3246-0/14/09…$15.00
http://dx.doi.org/10.1145/2664591.2664594 n
Jason Cohen
University of the Witwatersrand
Dept of Information Systems
Johannesburg, South Africa
27 (0)11 717-8164
Jason.Cohen@wits.ac.za
as MS Windows [6].
Leveraging off pointer-based interactions in designing gesture sets
could be justified [9]. The key motivation for doing so is that
users would be able to apply their existing knowledge of pointer-
based interactions when confronted with touch screen interaction.
An opposing view is that moving away from the classic WIMP
elements (windows, icons, menus, pointer) would improve the
quality of touch screen interactions [4]. According to this latter
view, multi-touch devices can enable exceptional freedom in
gestural input while the use of pointer-based, single finger, one-
point tap or drag interactions, analogous to mouse clicks and
drags, negates the benefits that multi-touch technology offers.
These conflicting views reflect an uncertainty about users’
preference between pointer-based interactions in touch screen
applications and novel interactions that are divorced from
traditional point-and-click interactions. The answer to the question
of which gestures would be most intuitive to users in different
application contexts may not have a simple answer.
An intuitive gesture is one where the gesture afforded by the
interface design is aligned to how users expect to provide input
[4]. Various factors, including the levels of previous exposure to
different interaction mechanisms, might influence what users
regard as intuitive. Therefore, understanding users and their
expectations is critical in designing appropriate touch screen
gesture sets.
Given the particular interest of this study in web searching on
multi-touch tablets, we asked the following question: What would
an intuitive user-defined gesture set for conducting an
informational search on a multi-touch tablet web browser look
like? In addressing this question, we developed a user-defined
gesture set based on gestures elicited from participants with little
or no experience with touch screen devices.
Next, we briefly discuss the related research. We then describe the
mechanisms used to engage individual users in the process of
reasoning, reflecting and articulating to construct the proposed
gesture set. We describe our data collection and analysis methods
and present our findings in the form of a gesture set together with
general guidelines for the design of intuitive gestures.
2. RELATED WORK
Previous research on gesture elicitation has predominantly
focused on tabletop devices. Some of the key studies are discussed
case-by-case below.
Nielsen, Störring, Moeslund and Granum [8] put forward a
procedure for developing an intuitive and ergonomic gesture set.
276
This procedure guides participants through the the process of
creating the gesture set, as well as testing it The study then
provides an example of how to apply the procedure through an
investigation of participants’ interactions with virtual 3D shapes.
The method used in our study was informed by this procedure.
Wobbrock, Aung, Rothrock and Myers [10] analysed participants’
unistroke gestures (these are gestures where the participant does
not lift his or her finger until the gesture is complete) on a
touchpad to indicate letters of the alphabet. They found that their
approach for developing gestures resulted in gestures that first-
time users were able to guess with considerable accuracy. Their
procedure also informed our study.
Epps, Lichman and Wu [1] investigated gestures used for general
operating system (Windows) tasks and identified users’
preferences for performing common Windows tasks using
gestures. They found, for example, that users prefer to use their
index finger and illustrated the need to reduce or remove the
Window-centric elements in screens presented to participants
during experiments.
Frisch, Heydekorn and Dachselt [2] created a user-defined gesture
set (comprising of one-handed and two-handed gestures, as well
as those involving hands and pen together) for editing node-link
diagrams on multi-touch, pen-enabled tabletops. They share their
procedure for creating this gesture set, which involved the use of
think-aloud data and requesting participants to provide ratings,
using Likert scales, for each gesture they propose.
Wobbrock, Morris and Wilson [11] studied users’ interactions
with 2D shapes (with no specific application context). They
present an approach to designing gestures that relies on eliciting
gestures from non-technical users through the use of referents.
Their findings indicated, among other things, that users rarely care
about the number of fingers they use when gesturing, that one
hand is preferred to two, and that desktop computing strongly
influences users’ mental models. This study greatly influenced
ours in terms of methodology, hypotheses and data analysis.
Micire, Desai, Courtemanche, Tsui and Yanco [6] conducted an
experiment to determine the gestures that people would naturally
use for controlling robot teams. They presented guidelines for
designing gesture sets, which we adopted in our experiments.
They also provided a taxonomy of user-defined gestures, which
provides the following classifications for gestures: Selection
gestures (used for selecting individual or multiple robots),
Position gestures (indicate desired change in location of the robot
or object), Rotation gestures (express the desired rotation of
robots, objects or map), Viewpoint gestures (used to change the
participant’s view by either moving the virtual world or changing
the position of the camera) and User Interface Elements (which
include commonly used interface elements like buttons, menus,
virtual keyboards and handwriting recognition).
Mauney, Howarth, Wirtanen and Capra [5] analysed gestures for
28 actions commonly done on handheld devices across
participants in nine different countries in order to identify cultural
similarities and differences in the use of gestures. Their findings
influenced the hypotheses of our study e.g. they found that
experience with gesture-enabled devices influenced the gestures
that participants created and that participants still relied on menus
when gesturing.
Kane, Wobbrock, and Ladner [3] conducted two user studies that
compared how blind people and sighted people use touch screen
gestures. Their first study showed that blind people have different
gesture preferences than sighted people and in a subsequent study
found significant differences in the speed, size, and shape of
gestures performed by blind people versus those performed by
sighted people.
It is worth noting that in the research surveyed above, only
Mauney et al. [5] and Kane et al. [3] addressed handheld
touchscreen devices, with the rest focusing on tabletop devices. It
should be further noted that the application contexts that have
been investigated have not specifically addressed the use of search
engines or web browsers for conducting any form of web search –
despite this being a common activity. These gaps in literature
provided justification for our study.
3. DATA COLLECTION
Users should play a central, participative role in gesture design,
because gestures designed autonomously by system designers are
not always reflective of what end-users would prefer [7, 11].
There are various ways in which users may participate in gesture
design. The approach employed in this study is based on the prior
work of Nielsen et al. [8], Wobbrock et al. [10], Wobbrock et al.
[11], Frisch et al. [2] and Micire et al. [6] who elicited gestures
from participants through a series of experimental tasks. The
approach taken is summarised below.
3.1 Referents for Relevant Functions
The first step in developing an intuitive gesture set for
informational web searching was to identify the functions for
which gestures would be needed [2]. An analysis of 80 web
browsers, comprising of both tablet and desktop web browsers,
yielded the 24 functions which were used in the study, and for
which referents were created. These referents are short video clips
(created using Mozilla Firefox, Windows Internet Explorer and
Google Chrome) illustrating the effects of the function i.e. the
output or effect that would be displayed when the function is
activated. The 24 functions are listed in Table 1.
3.2 Participants
Twenty participants, who were mainly university staff, served as
volunteers. Half were academics appointed at different levels
(from junior lecturers to full professors). The remainder consisted
of research assistants, administrative staff and a lab manager. One
participant was a business analyst. They represented a variety of
disciplines: computer science, information systems, linguistics,
sociology and business management. The inclusion criteria that
were used to recruit participants were:
 The required application domain knowledge, i.e. familiarity
with conducting an informational search on a desktop or
laptop web browser using the functionality detailed above in
Table 1.
 Little or no experience with any form of touch screen device,
including touch screen phones or tablets.
Participants’ ages ranged from 27 to 65, 6 participants were male
and 14 were female, and 17 were right-handed and 3 were left-
handed. The average age was 40. Eight had never used a touch
screen device before, while 12 had very limited exposure to touch
screen interaction.
277
 Table 1. Web browsing functions that support an During gesture elicitation, the researcher sat next to the participant
informational search and remained present throughout the session (see Figure 1). All
No Function No Function explanations and conversations that happened between the
researcher and the participant were recorded.
1 Go to previous 13 Paste section of text
webpage into search engine
2 Go to next 14 Open link on the
webpage webpage in new tab
3 Input search 15 Open a new browsing
query tab
4 Stop loading 16 Close a browsing tab
webpage
content
5 Refresh / reload 17 View record of
webpage visited pages
content
6 Pan down the 18 Delete entry from
webpage record of visited
webpages Figure 1 Experiment setting
7 Pan up the 19 Save current 3.4 Gesture Elicitation
webpage webpage address The referents were presented to participants on an iPad and the
within the browser participants were requested to propose gestures that they felt
for viewing at a later matched the referents – each one in turn. To ensure complete
stage understanding of the referents, the facilitator explained each
(“bookmarking”) referent to the participants prior to showing it to them. He
carefully explained the event(s) in the referent while showing it to
8 Pan across the 20 Call up saved them for the first time. If required, he showed it to them several
webpage webpage address times, addressing any questions the participants had, until they
9 Make webpage 21 Save image on indicated that they understood it clearly. The video referents of
content larger webpage to memory the web browser ensured that minimal elements relating to
(“zoom in”) Windows or the Macintosh were displayed; this was to avoid bias
[11].
10 Make webpage 22 Save current
content smaller webpage to memory When the participants were ready to offer a gesture or gestures, a
thin glass sheet was placed over the iPad. The referent video was
(“zoom out”)
then played on the iPad, and the participants performed the
11 Search for text 23 Request help gesture(s) for the event(s) in the video, as if interacting with the
on the webpage iPad. So, although the users’ gestures did not actually cause the
changes on the screen (because of the glass sheet that covered the
12 Copy a section 24 Close web browser iPad screen), the set-up very closely mimicked such real
of text on the interaction. The users performed the gestures on the iPad as if
webpage they were controlling the changes on the screen. Wobbrock et al.
[11] explain that with such an approach, the lack of feedback is
beneficial as it creates a “monologue” where all participants’ input
3.3 Experiment Setting is always acceptable, and therefore enables participants to exhibit
The experiments were conducted in a HCI laboratory equipped their natural, unrevised and uninhibited behaviour.
with two built-in, high-resolution, overhead cameras. In addition Participants were allowed to use one or more fingers, as well as
to these cameras, two standalone video cameras were positioned at one or both hands, any part or shape of their finger(s) and/or
the table where the participants’ gestures were elicited. All four hand(s) in any way they wanted, as long as they expressed the
cameras recorded the participants unobtrusively and were gesture which they felt would best matched the function shown in
positioned to optimally capture finger, hand and arm movements the referent [6]. Participants were requested to think aloud while
from different angles [2, 6, 8, 11]. providing their gestures [2, 6, 11]. These think-aloud data
A separate microphone was used to record the audio think-aloud provided verbal explanations and reasoning with regard to
data during the experiments. Noldus and Camtasia Studio participants’ gestures.
software were used to combine and synchronise the different As with Wobbrock et al. [10], if participants were unsatisfied with
video recordings and the audio data into a single video. the gesture they had proposed, they were allowed to redo it, but
once they had proceeded to the next gesture they were not

278
permitted to go back and change it. Where participants felt that an
additional widget, such as a menu or special button, was required
on the interface for a particular activity, they were allowed to
assume it was present – provided they explained in detail what the
widget was, the form they expected it to take and how they
expected it to work [6, 11].
3.5 Gesture Ratings
In addition to the gestures and think-aloud data, participants were
also requested to provide ratings, using Likert scales, for each
gesture they proposed. This was done immediately after
performing that gesture [2, 11]. These gesture ratings represented
the participants’ perception of the quality of their gestures in
terms of suitability and ease of performance and were elicited
using two Likert scales.
The first Likert scale, which elicited ratings for the suitability of
the proposed gesture for the referent displayed, read, “The gesture
I picked is a good match for its intended purpose” [11]. The
second scale, which elicited ratings for the ease with which the
proposed gesture could be performed, read, “The gesture I picked
is easy to perform” [11]. Five point Likert scales were employed,
both of which elicited ordinal responses from 1 = strongly
disagree to 5 = strongly agree. Obtaining the subjective ratings is
important as functions with low gesture ratings are functions that
users would likely struggle with – either in terms of their mapping
(reflected through the “gesture goodness” rating) or in terms of
how easy it is, physically, to perform them (reflected through the
“gesture ease” rating), or both.
4. DATA ANALYSIS
4.1 Classification of Gestures
The video data was evaluated in order to extract the gestures that
the participants used in their interactions. This was done by
extracting the gestures that each participant performed for each of
the referents. These gestures were then classified using the
Taxonomy of Surface Gestures [11], in corroboration with
categorisations used by [8] and [4]. The gestures were classified
according to 4 dimensions:
 Gesture Form: this relates to the fingers or parts of hand
used, whether the gesture was static (the finger(s) or hand is
in a fixed position over time) or dynamic (the finger(s) or
hand change positions over time) and whether it was a simple
(gesture comprises a single finger or hand action) or complex
gesture (gesture comprises two or more gestures, performed
sequentially or in parallel).
 Gesture Nature: this refers to whether the gesture is physical
(physically acts on objects), metaphorical (acts on, with, or
like something else), symbolic (visually depicts a symbol) or
arbitrary (arbitrarily mapped to referent).
 Gesture Binding: this refers to whether the gesture is object-
centric (performed with respect to a specific object), world-
dependent (takes the real world context into consideration),
world-independent (can generally be performed anywhere on
the interface) or has mixed dependencies (is world-
independent in one respect but world-dependent or object-
centric in another).
 Gesture Flow: this refers to whether the gesture is discrete
(performed in its entirety and once completely performed the
system provides feedback) or continuous (requires on-going
recognition by the system and feedback is provided while the
gestures is being performed).
Following this classification, the gestures were assigned to their
respective functions. This process is explained in the subsections
below.
4.2 Assignment of Gestures to Functions
Assignment of gestures to functions was based on agreement
between the different participants; that is, the gesture that had
been proposed the most number of times by different participants
for a specific referent would be assigned to that referent [10, 11].
It is important to note that the other gestures that were proposed
for that referent, i.e. those that were proposed less frequently,
were not discarded, but were also assigned to the referent
provided they did not conflict with other gestures [2, 4, 6, 11].
Three types of conflicts existed; conflicts within the participants’
own gesture sets, internal conflicts and external conflicts. These
are explained below.
4.2.1 Conflicts within Participants Gesture Sets
As with Wobbrock et al. [10] participants were required to resolve
conflicts among their own gesture sets, i.e. participants were
instructed not to propose the same gesture for different referents
where the interactive contexts was the same. For example, a
participant would not be permitted to use a single finger drag from
the left edge of the screen towards the middle of the screen for the
“Go to next page” referent and use the same gesture for the “Go to
the previous page” referent as the interactive contexts for these
referents are identical. The participant would, however, be
permitted to use that gesture when highlighting an entry from a
menu list that appears on the left edge of the screen.
4.2.2 Internal and External Conflicts
An internal conflict occurred when a participant’s gesture for a
certain referent conflicted with another participant’s gesture for
that same referent. External conflicts, on the other hand, occurred
when different participants used the same gesture, within identical
interactive contexts, for different referents.
To resolve any one of these types of conflicts between two
gestures, the gesture that had been proposed the most number of
times by different participants for that referent was assigned to
that referent and the one that was in the minority was excluded
from the final gesture set. While the conflicting gesture was
excluded, other gestures that did not have conflicts were still
considered for inclusion in the final gesture set for that referent
even if only proposed by a minority of participants. This was to
allow for alternatives in performing that function.
4.2.3 Agreement Scores
Once all the conflicts had been resolved, the agreement scores
were calculated. As previously mentioned, assignment of gestures
to functions was based on agreement between the different
participants; agreement scores are single numbers that serve as a
means of quantifying to what extent there was agreement between
participants for a gesture [11].
The computation of an agreement score was done by taking all
gestures that had been proposed for a specific referent, and based
on their classifications, grouping identical gestures together.
These groups, which represent the frequency with which the
279
different gestures were proposed by different people, were then
used to compute the agreement score A using the following
formula [11]:
where r is a referent in the set of all referents R, Pr is the set of
proposed gestures for referent r, and Pi is a subset of identical
gestures from Pr. Using the formula to calculate agreement scores
is important because the greater the agreement between different
participants in the gestures used for a referent, the more likely it is
that other novice users would also agree or subscribe to using
those gestures. Those functions with low agreement scores, are the
functions that users would likely struggle with their gestures.
4.3 Think-Aloud Data
The physical form of a gesture, i.e. the gesture as it is physically
performed, may not always reveal the nature of the gesture. The
think-aloud data, which was participants’ narrations, explaining
their thought processes for the gestures they were proposing while
they were simulating the gestures, were used in revealing the
nature of these gestures. It assisted in revealing the participants’
mental models within the context of the referent, assisting us in
the classification of the gesture as previously described.
4.4 Analysing the Gesture Ratings
Based on the subjective Likert scale ratings provided by each
participant for each gesture proposed, average “gesture goodness”
Table 2. Web browsing functions and their gestures
Function Gesture(s)
1. Go to
previous
webpage
The user taps on the “Back” button.
2. Go to next
webpage
The user taps on the “Forward” button.
ratings (based on the scale of “The gesture I picked is a good
match for its intended purpose”) and “gesture ease” ratings (based
on the scale of “The gesture I picked is easy to perform”) were
calculated [11]. These data provided crucial complementary data
to the actual elicited gestures and provided an opportunity to
evaluate the extent to which participants adjudged their gestures to
be suitable for the referents they had proposed them for, as well as
how easy they considered them to be to perform.
The gestures with low gesture ratings are functions that users
would likely struggle with – either in terms of their suitability for
the function they were proposed for (reflected through the
“gesture goodness” rating) or in terms of how easy they are,
physically, to perform (reflected through the “gesture ease”
rating), or both. No gestures had to be removed from the gesture
set as a result of low ratings. The lowest rating across all
participants for both Likert scales was 3, with the averages for the
“gesture goodness” and “gesture ease” ratings being 4.494 and
4.527 respectively.
5. FINDINGS
5.1 The Gesture Set
A coherent gesture set for conducting an informational search on a
multi-touch tablet web browser, based on gestures elicited from
participants, was successfully created. Table 2 presents the 24
search functions and the gesture(s) found to be most suitable for
those functions. It should be noted that the gestures presented
below are only a subset of the total gesture set. Note that the
buttons that appear on the interfaces in the images below The full
set can be requested from the authors.
Function Gesture(s)
13. Paste
section of text
into search
engine
14. Open link
on the
webpage in
new tab
The user drags the link to the “New Tab
Button”.
280
3. Input search 15. Open a
query new browsing
tab

The user calls up a keyboard by tapping

once on the Google textbox. The user taps once on the “New Tab
Button”.

4. Stop 16. Close a

loading browsing tab
webpage
content

The user taps once on a “Stop” button. The user taps once on the “Close Tab”
cross.
5. Refresh / 17. View
reload record of
webpage visited pages
content

The user taps once on a “Refresh / The user taps once on a “History Button”
Reload” button.

6. Pan down 18. Delete

the webpage entry from
record of
visited
webpages

The user drags their finger downwards The user by taps once on the item, and
along the right edge of the screen. taps once on a “Delete Button”.

281
7. Pan up the 19. Save
webpage current
webpage
address within
the browser
for viewing at
a later stage

The user taps once on the “Favourites

The user drags their finger upwards
Button” to call up a list of webpages that
along the right edge of the screen.
have been saved and taps once on the
“Add Button”.
8. Pan across 20. Call up
the webpage saved
webpage
address

The user taps once on the “Favourites

The user drags their finger sideways Button” to call up a list of webpages that
(left to right) along the bottom edge of have been saved and taps once on the
the screen. page to be opened.

9. Make 21. Save

webpage image on
content larger webpage to
(“zoom in”) memory

The user draws a box around the image

The user makes the webpage content by dragging the finger in a rectangle
larger by dragging his / her thumb and around the image, and taps once on the
index finger apart. “Save Button”.

10. Make 22. Save

webpage current
content webpage to
smaller memory
(“zoom out”)

The user makes the webpage content The user calls up the main menu by
smaller by dragging his / her thumb tapping on a menu button, and taps once
and index finger together. on the “Save webpage” option.

282
 11. Search for
text on the
webpage
The user calls up a textbox by tapping
on a “search button”.
12. Copy a
section of text
on the
webpage
The user presses with a single finger
where the text begins, and drags the
finger over the text.
This gesture set revealed the following:
Co-existence of One-Handed and Two-Handed Gestures
It was found that users prefer one-handed gestures to two-handed
gestures. Of the 478 gestures that were recorded, 459 (96.025%)
were one-handed, and 19 (3.975%) were two-handed. This
implies that:
 The use of two-handed gestures should be limited.
 If two-handed gestures are developed for a specific function,
an alternative one-handed gesture should be made available.
It is also worth noting that during the experiments, participants
did not actually hold the iPad in their hands – it was placed on a
table, facing participants at an incline. During normal use a tablet
would be held in one hand, while the other hand gestures. Under
those circumstances it would be difficult to use two-handed
gestures. Our participants had both hands available for gesturing
and still opted to use one-handed gestures.
The only notable exceptions were the referents for resizing (i.e.
zooming in and out).
Inclusion of Desktop Computing Elements
We found that participants relied heavily on desktop computing as
a point of reference for their gestures. The single finger tap,
analogous to the mouse click, was the most used gesture and
accounted for 50.727% of all the gestures used. The single finger
drag, analogous to the mouse drags, was second in line, and
accounted for 24.128% of all gestures performed. Furthermore,
the majority of participants incorporated the use of menus (main
menus and context menus) into their gestures. The following
implications arise:
23. Request
help
The user taps once on a “Help” button.
24. Close web
browser
The user taps once on a “Close Button”
located at the top right corner of the
screen.
 Novel gestures for tablet web browsing that do not
correspond to any of the ways in which desktop web
browsing is performed, may not be intuitive to adult users
who had mostly been exposed to mouse based interaction.
 Should novel gestures be developed for tablet web browsing,
alternative gestures that are desktop computing adaptations,
including those that require menu access, should be made
available to users.
A key observation is that users drew upon their experiences of
desktop computing despite the exclusion of graphical elements
relating to Windows or Macintosh operating systems (these
included menus, title bars and buttons such as Back and Refresh).
For example, 14 out of 20 participants tapped on an imaginary
“X” in the top right-hand corner of the screen for closing the web
browsers despite the fact that none of the referents had the
Minimise, Maximise and Close buttons typically found on the top
right corner of windows.
Another observation was that participants generally dragged their
fingers from the top to the bottom to scroll down, and from the
bottom to the top to scroll up. On most tablets, the opposite is true
i.e. users drag their fingers from the top to the bottom to scroll up
and from the bottom to the top to scroll down. The fact that the
Windows operating system generally employs arrow keys or
scroll bars explains why it is that most participants (13 out of 20)
in this study dragged or flicked downwards on the screen to scroll
down the webpage, and vice versa.
The complexity introduced by adopting a desktop computing
adaptation in this situation, and allowing users to drag downwards
in order to scroll down, means that users that have become
accustomed to dragging/flick upwards to scroll down will find
this method of scrolling unintuitive. One recommendation in this
283
situation would be to allow users to decide which outcomes
should be produced when dragging downwards or upwards by
allowing users to configure what function should be mapped to
which gesture
A major disadvantage of desktop computing adaptations is that
they may not make the best use of multi-touch technology,
especially given the predominant use of single finger taps. The
advantage of this approach, however, is that users can move
between their desktops, which they are familiar with, and their
tablet devices without having to learn or memorise new ways with
which to interact with the tablet web browser.
The use of desktop computing adaptations should, however, not
overshadow the fact that several participants produced novel
gestures, such as dragging a finger repeatedly over an item to
delete it, as if “rubbing it out” (Referent 18), dragging a browsing
tab “off the screen” to close it (Referent 16) and tracing a
question mark on the screen to activate the “Help” function
(Referent 23), to mention a few. These were not excluded from
the gestures set, but incorporated to co-exist with gestures that are
desktop computing adaptations.
5.2 Gesture Re-use
It was found that participants will attempt using the same gesture
to accomplish different tasks (in different interactive contexts)
and re-use the same gestures in different combinations based on
their judgment of how tasks should be decomposed. This
reinforces the existing knowledge that users differ in how they go
about completing various tasks. For designers and developers it
means that gesture re-use is essential and should be maximised,
while obviously ensuring that no conflicts occur. Our data
revealed five gestures that should receive special consideration in
terms of re-use. They are: the one finger tap, the one finger drag,
the one finger flick, the one finger press and the one finger
double-tap. These accounted for 86.065% of all gestures provided
by participant across all referents.
6. LIMITATIONS
The depth of our data collection and consequently the amount of
time spent with each participant (and also the time required for
analysis of each participant’s data) restricted the number of
participants to 20. The sample size proved large enough to inform
an intuitive gesture set, but we could not analyse the gestures in
relation to the demographics of the participants, e.g. identifying
whether there were any patterns in the gestures provided by male
participants in comparison to female participants, or between
gestures proposed by left-handed versus right-handed people.
The study addressed 24 specific functions. This means that while
the user-defined gesture set addressed the key functions for
conducting an information search, it does not cover the full
complement of web browsing functions.
7. CONCLUSIONS AND FUTURE WORK
The study has achieved its aim of creating a user-defined gesture
set for conducting an informational search on a multi-touch tablet
web browser based on gestures elicited from participants. This
user-defined gesture set may be used as presented here by
designers or developers in new versions of web browsing software
for multi-touch tablet devices. Designers may also decide only to
use certain gestures, or to modify the proposed gestures as they
deem fit for their applications. Furthermore, while these gestures
have been created specifically for conducting an informational
search on a multi-touch tablet web browser, it is not inconceivable
that designers may employ some of the gestures presented in this
study in other application contexts. Functions such as “searching
for text” or “request help” are not limited to web browsing
application context.
In addition to the user-defined gesture set, this study provided
various insights with respect to gesture design for multi-touch
tablet browsing. These enhance the understanding or appreciation
of what users, specifically adult users who have had limited or no
prior experience with touch screen devices, consider to be
intuitive. This understanding will enable designers to design
gestures that are aligned to the naïve or natural expectations of
their users, and would therefore result in satisfactory or enjoyable
experiences for these users – potentially attracting more users to
their applications. The contributions of this study will therefore
prove beneficial to those designing or developing multi-touch
tablet interfaces.
There are various implications for future research. An obvious
next step would be to conduct a usability study on the user-
defined gesture set, using completely different participant groups,
to determine the extent to which the gestures presented in this
study are usable in practice.
It is recommended that similar studies be conducted for tablet
devices, but focusing on other application contexts, such as word
processing, note taking, viewing videos, playing music and
gaming. With tablet devices becoming more popular within the
public domain, it becomes increasingly important to ensure that
usability requirements for the various activities that tablet devices
support, are well understood by those in practice.
8. REFERENCES
[1] Epps, J., Lichman, S., and Wu, M. (2006). A Study of Hand
Shape Use in Tabletop Gesture Interaction. In Proceedings of
the 2006 Annual Conference on Human Factors in
Computing Systems – CHI ’06, Montreal, Canada, 22 – 27
April.
[2] Frisch, M., Heydekorn, J., and Dachselt, R. (2009).
Investigating Multi-touch and Pen Gestures for Diagram
Editing on Interactive Surfaces. In Proceedings of the ACM
International Conference on Interactive Tabletops and
Surfaces – ITS ‘09, Calgary, AB, Canada, 23 – 25
November.
[3] Kane, S. K., Wobbrock, J. O., and Ladner, R. E. (2011).
Usable Gestures for Blind People: Understanding Preference
and Performance. In Proceedings of the 2011 Annual
Conference on Human Factors in Computing Systems – CHI
’11, Vancouver, Canada, 7 – 12 May.
[4] Lao, S., Heng, X., Zhang, G., Ling, Y., and Wang, P. (2009).
A Gestural Interaction Design Model for Multi-touch
Displays. In Proceedings of the 23rd British HCI Group
Annual Conference on People and Computers: Celebrating
People and Technology – BCS-HCI ’09, Cambridge, United
Kingdom, 1 – 5 September.
[5] Mauney, D., Howarth, J., Wirtanen, A., and Capra. M.
(2010). Cultural Similarities and Differences in User-
Defined Gestures for Touchscreen User Interfaces. In
Proceedings of the 28th ACM CHI Conference on Human
Factors in Computing Systems, Atlanta, USA, 10 – 15 April.
284
[6] Micire, M., Desai, M., Courtemanche, A., Tsui K. M., and
Yanco, H. A. (2009). Analysis of Natural Gestures for
Controlling Robot Teams on Multi-touch Tabletop Surfaces.
In Proceedings of the ACM International Conference on
Interactive Tabletops and Surfaces – ITS ‘09, Calgary, AB,
Canada, 23 – 25 November.
[7] Morris, M. R., Wobbrock, J. O., and Wilson, A. D. (2010).
Understanding Users' Preferences for Surface Gestures. In
Proceedings of Graphics Interface 2010 – GI '10, Toronto,
Ontario, Canada, 31 May – 2 June.
[8] Nielsen, M., Störring, M., Moeslund, T.B., and Granum, E.
(2003). A procedure for developing intuitive and ergonomic
gesture interfaces for HCI. In Proceedings of the 5th
International Gesture Workshop 2003, Genova, Italy, 15–17
April.
[9] Wang, F., Cao, X., Ren, X., and Irani, P. (2009). Detecting
and Leveraging Finger Orientation for Interaction with
Direct-Touch Surfaces. In Proceedings of the 22nd annual
ACM symposium on User Interface Software and
Technology – UIST ’09, British Columbia, Canada, 4–7
October.
[10] Wobbrock, J.O., Aung, H.H., Rothrock, B., and Myers, B.A.
(2005). Maximizing the guessability of symbolic input. . In
Proceedings of the 2005 Annual Conference on Human
Factors in Computing Systems–CHI '05, Portland, Oregon,
2–7 April.
[11] Wobbrock, J. O., Morris, M. R., and Wilson, A. D. (2009).
User-defined gestures for surface computing. In Proceedings
of the 27th international conference on human factors in
computing systems – CHI 2009, Boston, Massachusetts,
USA, 4 – 9 April.
285