C

O

V

E

N

T

R Y

U

N

I

V E

R

S

I

T

Y

Faculty of Engineering and Computing Department of Computing and the Digital Environment

Creative Computing 303CDE - Individual Project

Guidelines for Gestural Tablet Interaction Author: Thomas Davies SID: 2179955 Supervisor: John Halloran
Submitted in partial fulfillment of the requirements for the Degree of Bachelor of Creative Computing

Academic Year: 2010/11

Declaration of Individuality
This project is all my own work and has not been copied in part or in whole from any other source except where duly acknowledged. References list. I agree that an electronic copy of this report may be stored and used for the purposes of plagiarism prevention and detection. I understand that cheating and plagiarism constitute a breach of University Regulations and will be dealt with accordingly. As such, all use of previously published work (from books, journals, magazines, internet, etc) has been acknowledged within the main report to an entry in the

Signed:

Date:

Office Stamp

Copyright
The copyright of this project and report belongs to Coventry University. Declaration of Individuality i

Abstract
The tablet devices market is growing substantially and it expected to sell 140 million devices in 2014 and therefore there is a need for interaction guidelines that span both applications and platforms. This report presents a set of guidelines for gestural tablet interaction based on experienced and inexperienced participants who were asked to create gestures for common computer actions. 18 participants created 324 gestures for 18 actions; gestures that were analysed and categorized before being rated on their appropriateness for the intended action. This data provided the foundation of the set of guidelines for gestural tablet interaction that I created and were then validated by another set of users.

Abstract

iii

Table of Contents
Additional Materials on the CD Acknowledgements Introduction & Background
Background to the Project Recent History Market Potential Method of Study Findings The Need for a Gesture Set Conclusion Section Summary

viii ix 1
1 1 2 2 3 4 5 6

Methodology
Introduction Researching Ethics Guessability Testing Inter-Rater Agreement Gesture-Meaning Association Testing Section Summary

7
7 7 7 11 12 14

Formative Investigation
Introduction Guessability Study Inter-Rater Agreement Section Summary

15
15 15 15 17

Findings
Introduction Guessability Study Rating Study Table of Contents

19
19 19 22 v

Section Summary

25

The Gesture Set
Introduction Guidelines for Gestural Tablet Interaction Section Summary

27
27 27 32

Summative Investigation
Introduction Gesture-Meaning Association Testing Alterations to Original Methodology Results Findings Section Summary

33
33 33 34 34 35 37

Discussion & Conclusion
Conclusion Value & Benefit Implications Challenges Future Work

39
39 39 40 40 40

Project Management
Supervisor Meetings & Progress

43
43

References Bibliography Appendix A
Guessability Study Documentation & Forms

45 47 49
49

Appendix B
Gesture Screenshots

85
85

Appendix C
Rating Study Documentation & Forms

123
123

Appendix D
vi Guidelines for Gestural Tablet Interaction

127

Gesture Data Charts

127

Appendix E
Gesture-Meaning Association Results

137
137

Appendix F
Supervisor Correspondence

139
139

Table of Contents

vii

Additional Materials on the CD

viii

Acknowledgements
‘Technology alone is not enough’ Steve Jobs

This thesis would not have been possible without the support and guidance of Dr John Halloran who has assisted this project from the very beginning, through his enthusiasm and immense knowledge. I would also like to express my gratitude to all the Creative Computing staff who have played an instrumental role in my development over the past three years. Finally, I wish to thank my parents who have always supported me and ensured that I had every chance of fulfilling my dreams.

Acknowledgements

ix

1. Introduction & Background
1.1. Background to the Project
The launch of the iPad in early 2010 ushered in a new class of tablet devices that are radically different than the earlier tablet PCs of the 1990s and early 2000s. Styli and desktop-based operating systems have been replaced by multi-touch displays and mobile operating systems. After 40 years of using the same human-computer interaction paradigms created by the infamous Xerox PARC team of the 1960s and 1970s, change has come in the form of light, compact, and efficient modern tablet devices (Saffer 2008: 3). This shift in paradigm is different however to that of the desktop era where only two major platforms led the way - Windows and Mac OS - as multiple companies have ventured into the tablet sector. Users will be introduced into this new method of interaction via a fragmented sector in both hardware and software and therefore it is important to define a set of multi-platform guidelines and standards as the methods we create today will define the landscape for decades to come (Saffer 2008: 4). 2011 will be a defining year for tablet devices as many manufacturers will be launching their products, aiming to eradicate Apple's huge market share.

1.2. Recent History

HP WebOS

HP TouchPad announced Feb 2011 Playbook announced Sep 2010

HP TouchPad released Summer 2011 Playbook released Apr 2011

RIM

Android

Galaxy Tab released Nov 2011

Motorola Xoom released Feb 2011 Honeycomb OS released Feb 2011 iPad 2 released Mar 2011

Galaxy Tab 10 released Summer 2011

Apple

iPad released Apr 2010

January 2010

April 2010

July 2010

October 2010

January 2011

April 2011

July 2011

October 2011

Figure 1.1 Timeline of recent history in the tablet sector In September 2010, Research in Motion (RIM) introduced their rival to Apple's iPad; the PlayBook. The OS will not be based on RIM's smartphone operating system but on QNX Software Systems' OS which 'is designed specifically for the tablet size computer' (BBC News 2010). QNX was acquired by RIM in April 2010, 5 months before the PlayBook announcement (RIM 2010). The PlayBook -

Chapter 1: Introduction & Background

1

designed for the enterprise sector - is expected to be 'commercially available early in 2011' (BBC 2010). While Samsung released their Galaxy Tab in November 2010, it was using an Android build that was designed for smartphones and not tablets and therefore cannot be considered as a legitimate tablet device. Therefore, throughout 2010, Apple dominated the sector with a 'market share in tablet computers at 95 percent in the third quarter' (Bloomberg 2010). The next major tablet announcement took place on February 2nd, 2011 when Google announced the Honeycomb update to their Android OS that was targeted specifically for the tablet form factor (Google 2011). This Google update then resulted in several manufacturers; Samsung, Motorola and Asus announcing that their Honeycomb-based tablets will launch in the first half of 2011. One week after the Google announcement, HP revealed plans for their recently acquired Palm webOS mobile operating system. Along with updates to their smartphones, HP introduced the TouchPad; a webOSbased tablet device that will launch in Summer 2011 (Palm 2011). On March 2nd, 2011 Apple announced the second iteration of the iPad and that it would be available in the North American market on March 11th and on March 25th for the international market. Apple announced their second tablet device significantly later than all competitors, yet were able to launch the product significantly before all other manufacturers - except Motorola with their Xoom tablet. Therefore, 2011 will see four mobile operating systems vying for market share; Apple's iOS, webOS, Android and RIM's Tablet OS. While three of those OSs are not licensed to run on third-party hardware, Android is open-source and is currently used by several hardware manufacturers; fragmentation will occur on both the hardware and software aspects of tablet devices.

1.3. Market Potential
The iPad became commercially available on April 3rd, 2010 and sold its millionth device 28 days after launch. Comparatively, it took 73 days for the original iPhone to reach the 1 million mark. (Yahoo News 2010). By the end of 2010 - 9 months since launch - Apple had sold over 14 million devices (Apple 2011). Meanwhile, research firms 'forecast 57 million tablets will be sold' in 2011 and expects the sector to reach 171 million devices in 2014 (Fox Business 2011) amounting to one in five adults owning a tablet device (Palm: 2011).

1.4. Method of Study
The fundamental basics of both the International Usability Partners and the University of Washington’s study consists of this ‘guessability study methodology’ (Wobbrock et al 2009: 1). As the University of Washington used live hardware, users were shown ‘the effect of gestures’ via an onscreen animation and then asked to perform a gesture to invoke that effect. However, the International Usability Partners study utilised a ‘lo-fi prototype simulating a touchscreen’ with before and after screenshots, asking users how they would produce what was on the after screenshot (Mauney et al 2010: 2) as to ‘focus on the user’s intuitive behaviour’ (Mauney et al 2010: 13). Dan Saffer also agrees that it is important to ‘employ the knowledge and intuition of those who will use it’ 2 Guidelines for Gestural Tablet Interaction

and similarly suggests using this method, or reversing it by demonstrating ‘a gesture and see what feature users would expect that gesture to trigger.’ (Saffer 2008: 28). For the specifics, both studies chose to remove all bias by having ‘no elements specific to Windows or the Macintosh’ while ensuring that ‘the commands were application-agnostic’ (Wobbrock et al 2009: 2). This is important because the studies wanted to limit the amount of gestures that were learnt from previous experiences rather than being purely intuitive. The University of Washington were concerned about the lack of research and that ‘gestures have been defined by system designers’ and not users, therefore they believe that it is right to ‘consider the types of gestures people make without regard for recognition or technical concerns.’ (Wobbrock et al 2009: 1). While the International Usability Partners study selected a range of participants, both experienced and inexperienced the University of Washington study only ‘recruited non-technical people without prior experience using touch screens’ as they felt this was the only subset of people that ‘would behave with and reason about interactive tabletops differently than designers and system builders.’ (Wobbrock et al 2009: 2). Therefore, it was interesting to note that the International Usability Partners research found it difficult - even with inexperienced users - to understand when a participant performed a gesture that was naturally intuitive due to ‘local and media exposure of touchscreens.’ (Mauney et al 2010: 12).

1.5. Findings
The International Usability Partners study found that on tablet-sized devices there was ‘frequent usage of the ‘tap’ gesture’ suggesting ‘participants imagined a button or context menu to trigger the required action with a tap.’ (Mauney et al 2010: 10). Their study produced different results to the University of Washington’s research into surface computers where ‘users rarely care about the number of fingers they employ’ (Wobbrock et al 2009: 1). While tablet devices and surface computers differ in size, tablet devices are large enough to take several inputs at once similar to surface computers. An issue with the International Usability Partners report is that is doesn’t specifically state how the proposed gesture set was developed. Questions remain as to whether it was solely based on the analysis of their user studies, or a combination of that and a designer’s perspective. Yet, the University of Washington clearly states that the final gesture set ‘was developed in light of the agreement participants exhibited in choosing gestures for each command.’ So, if ‘more participants used the same gesture for a given command, the more likely that gesture would be assigned to that command’ and the study actually found that the ‘user-defined gesture set emerged as a surprisingly consistent collection founded on actual user behavior’ (Wobbrock et al 2009: 2). While the International Usability Partners study was purely quantitative; producing only data from the gesture testing, the University of Washington used ‘Likert scales concerning gesture goodness and ease’ (Wobbrock et al 2009: 3) and utilised the think aloud method to gauge how users were feeling when creating the gestures.

Chapter 1: Introduction & Background

3

1.6. The Need for a Gesture Set
While the likes of Apple and Google have interface guidelines that they advise third-party developers to follow, the majority of the guidelines focus on the aesthetics of the application and not the interaction techniques. In 2000, Jef Raskin stated that modern computers were 'a feast for the eyes, with glorious colorful backgrounds and three-dimensional tromp l'oeil', a fact still true for modern tablet devices. Yet while aesthetics play a role in the overall experience, 'the quality of any interface is ultimately determined by the quality of the interaction between one human and one system' (Raskin 2000: xix). 1.6.1. Habituation After prolonged and persistent practice, people will habitually complete a task without consciously thinking about it. Therefore the gestures that are developed for tablet devices will become habit for users, whether they are good or bad. The issue with developing habit for bad practices is that 'you cannot break a habit by an act of violation' but simply by 'a time-consuming training process' (Raskin 2000: 20). It is therefore pivotal that the correct gestures are learnt by users otherwise they will struggle to habituate to another set of gestures. 1.6.2. Inconsistent Gesture Usage Many current iPad apps are creating different methods to complete a similar action, yet this does not benefit the end user as 'having multiple options can shift your locus of attention from the task to the choice of method' which drastically hinders the user's experience of the product (Raskin 2000: 20). Altering a method for an action can act as an unique selling point and is actively promoted in advertising as shown by Apple's original iPhone adverts that highlighted key interaction styles and techniques in Figure 1.2.

Figure 1.2 Apple’s ‘How to’ iPhone advert (Apple 2007) While this differentiates manufacturers from each other, the practice is detrimental to users as 'inconsistent gesture usage can lead to frustration, confusion and usability problems' (Ginsburg 2010: 22).

4

Guidelines for Gestural Tablet Interaction

1.7. Conclusion
When discussing the need for gestural standards, Dan Saffer imagines a scenario where physical movements would be required to turn on the light in a darkened room. With no indication about what gesture is required, you attempt several random movements in an attempt to turn on the light. While this might be an 'imaginable future scenario' for free-form gestural interfaces, these sorts of scenarios are already appearing for touch-based systems. Dan Saffer suggests setting up 'some sort of standards board, similar to what the Web has with the W3C', a board that has helped define the future of the web across a multitude of devices. But, if we do not work with the individual companies - such as Apple, Google and HP - they will simply continue to create gestures and try and force them onto end users as standards. The concern on individual companies leading the charge for gestural standards is the possibility of them patenting their creations. Saffer states that the actual gesture probably can't be patented as it fails two US patent requirements; it's natural and too obvious. However, Apple is currently attempting to patent 'a gesture tied to a specific action' - specifically the Pinch to Shrink gesture - and if this is accepted, then they will be able to license the patent or force infringing parties to stop using the gesture. This would then force other manufacturers to create their own gestures so that they won't infringe on others' patents and will then look to protect these gestures in retaliation. While Saffer attempts to reassure us by stating that 'patent holders do not have to enforce their patents', they still have the option of suing infringing parties in their arsenal. Ctrl + C is the standard command to copy an item in Windows and it is shared by all third-party Windows developers; the command is even similar on Mac OS X with Command + C (Where 'Command' replaces 'Ctrl' for all commands on Mac OS X). With further patent claims by tablet manufacturers and operating system developers, users will have to learn 'new commands for actions such cut-and-paste for every program' and platform (Saffer 2008: 174-176). If we don't act soon, users will be unable to go from platform to platform and utilise their habitual knowledge as they will have to 'remember a crazy number of gestures for common actions' (Saffer 2008: 174). We are on the brink of a new computing paradigm which could radically change how people interact with technology and therefore we should create a set of standard gestures that can work across different platforms and applications.

Chapter 1: Introduction & Background

5

1.8. Section Summary
1.8.1. The Market • The iPad was launched in 2010 and this spurred many other manufacturers to produce their own competing tablet devices. • Major companies launching tablet devices in 2011 include Apple, HP, Motorola, RIM, Samsung and Acer. • The tablet market is very different to the early desktop computing landscape. Multiple companies have entered the tablet market, and the dominance of a couple of platforms seems unlikely. • Industry experts expect the fledgling market to grow substantially over the next 5 years, with the sector selling over 150 million tablet devices in 2014. 1.8.2. Prior Research • Tablet devices received little exposure in academia. • Previous academic research is limited to touch-based phones and surface computers. • Research involved real users and potential users throughout the entire study. • They noted the lack of user involvement in current gestural creation. 1.8.3. The Need for Guidelines • Gestures used by the end users will become habit, whether they are good or bad practice. • Breaking a habit is challenging and can only be achieved by a time-consuming process. • Inconsistent gestures have already appeared across both applications and platforms. • Manufacturers and software developers are keen to protect their intellectual property and have begun patenting their creations and suing infringing parties. Action/gesture relationships are a key issue in these lawsuits.

6

Guidelines for Gestural Tablet Interaction

2. Methodology
2.1. Introduction
This section discusses the methodology for the entire study, detailing the processes behind the three stages of creating a set of guidelines for gestural tablet interaction. As shown in Figure 2.1, the individual stages lead on and inform the proceeding stage.

Guessability Study

Inter-Rater Agreement
Figure 2.1 The 3 stages of the research

Gesture-Meaning Association Test

2.2. Researching Ethics
Collecting information from the public can be challenging as they are often concerned about how their data will be used. Therefore, it is pivotal that I handle their data with care; informing them that it will be used anonymously. Moreover, making participants more comfortable benefits me as the researcher because ‘it will yield better results, since the subjects will likely open up more if they know that they (and their data) are being treated well.’ (Saffer 2007: 74).

2.3. Guessability Testing
2.3.1. Process
Guessability Test Recruit Particpants Planning Create Discussion Guide Think Aloud Protocol Observation & Questioning Data Analysis Presentation of Data

Figure 2.2 Guessability testing process 2.3.2. Guessability Method Prior to undertaking the test, the participants completed a consent form that confirmed they were willing to participate in the study and that the results from their tests will be used in the report anonymously. Following this, they completed a short questionnaire that documented their demographics in terms of age and gender along with information about their experience using tablet devices. While there was no specific age range requirements, it will be interesting to see how different subsections of people could possibly create different gestures. The results from questioning the users’ experience would ensure I have a 50/50 split; half experienced and half inexperienced. Chapter 2: Methodology 7

When conducting the actual tests, subjects were shown a short animation - as shown in Figure 2.3 that displayed the outcome of the gesture for that action. To ensure that the participants were not guided to perform a specific gesture, the animations were single slides and lacked any animation. For example, for the Zoom-in action, the video didn’t slowly transition the smaller squares into larger squares as this could result in the users performing a gesture based on the video and not based on intuition.

Slide 1 - Before Screenshot

Slide 2 - After Screenshot

Figure 2.3 Zoom-in action slides for video This approach is similar to the University of Washington’s study where after the animation, the participants were shown ‘the objects as they were before’ the action was performed and then they were expected to perform the gesture they felt would invoke that desired action (Wobbrock et al 2009: 4). The before screenshot was placed as the background of a drawing app - called Doodle Buddy - on the iPad that allowed for multi-finger input and following the drawing of the gesture, I would take a screenshot of the canvas. After each gesture, the participants were asked to rate their created gesture based on the statement: ‘The gesture I picked is a good match for its intended purpose’ and rated it on a 5-point likert scale from Strongly Disagree to Strongly Agree. The rating along with the recorded video of the participants performing the gestures and screenshots - will be used in analysing the results from this test. 2.3.3. Participants For most usability studies, Jakob Nielsen recommends 5 users as they find 90% of issues. This study however, isn’t a typical usability study and therefore there isn’t enough data available to set a precedent. The International Usability Partners study used 40 participants per country, with 20 experienced and 20 inexperienced users. They gave no reason as to why they selected that number and I also feel that 40 participants are too many for my project due to time and resource constraints. I attempted to find similarities between this method and other types of usability studies. Card sorting seems similar as it looks for similarities in user’s decisions with the aim of producing a taxonomy and gaining understanding of their mental models. For card sorting, Nielsen suggests 15 users. As I also have two types of users; experienced and inexperienced I would need around 30. 8 Guidelines for Gestural Tablet Interaction

When testing multiple categories, you don’t need to test 15 users per group as there will be overlaps in the results. As Nielsen discussed the 5 person rule for most usability studies, he stated that you only need 3 users per category when testing 2 groups. That amounts to 60% of the original 5 person rule. Therefore, the recommendation would be to use 18 participants; 9 experienced and 9 inexperienced. Usually, the selection ‘of participants whose background and abilities are representative of your product’s intended end user is a crucial element of the testing process,’ yet one main aim of this study is to discover how outcomes from experienced and inexperienced users differ (Rubin 1994). Moreover, the modern tablet market is still maturing and there is no knowledge of what specific types of people currently use tablet devices and who will in the future. 2.3.4. Equipment & Materials
Device iPad Panasonic Lumix TZ7 Usage For showing the short animations and for completing all the guessability testing For recording the participants performing the gestures

Table 2.1 Set of actions to be used in the Guessability Testing For the study, the participants will be using an Apple iPad. Unlike the International Usability Partners study which utilised webcam rig that recorded the piece of paper that was placed on an immovable object, I feel it is important to allow the user to fully embrace a tablet by being able to grasp and hold the physical object in a way that is natural to them. By making participants feel more relaxed by allowing them to use the tablet, the results extracted from the study will be more reliable. I will use an iPad application that allows me to place an image - for this study; a screenshot - in the background and then ask the participant to enact a gesture that they feel will accomplish the given task. 2.3.5. Environment To further allow participants to relax I will use a guerrilla usability testing style. One method is known as coffee shop testing where ‘researchers go to their local coffee haunt in search of study participants.’ (Ginsburg 2010: 181) This method is also beneficial to me as I lack the big budget required to have a usability lab and entering such labs can often be a daunting task for any participant. The consumer electronics companies are advertising these products with relaxing imagery; lounges and living rooms with the users resting their feet on a table. I aim to mimic this calm environment to ensure my results are reliable. 2.3.6. Think Aloud Protocol The guessability method yields a lot of quantitative data, however to fully understand users’ mental models, qualitative data is necessary. During the studies, participants were asked to think aloud as this will allow us to ‘obtain rich qualitative data that illuminates user’s mental models’ (Wobbrock et al. 2009: 1). Utilising this method allowed us to have participants ‘express their confusion, frustration and even their delight’ as well as discuss their thought process and reasoning behind the gestures .they were performing (Rubin 1994: 218). Moreover, it also allowed me to ask more informed questions after each gesture as I had a basic understand as to why they felt that gesture should invoke a specific action. Chapter 2: Methodology 9

2.3.7. Questioning Working in collaboration with the think aloud protocol, questioning participants allowed us to ‘fully understand the “why” behind performance and preference’ (Rubin 1994: 219). I also ensured that I was asking open-ended questions to ‘elicit more and better information’ than closed-ended questions as they can suggest to the participants that you are only looking for a limited answer that rarely yields enough qualitative information. Moreover, the open-ended questions ‘help prevent “leading the witness”’ which is especially critical when creating a set of guidelines for interaction (Goodwin 2009: 60). 2.3.8. Set of Actions Based on the International Usability Partners and University of Washington studies, along with Dan Saffer’s Designing Gestural Interfaces, I selected 18 actions that are required for every day computational tasks. The 18 actions are shown in Table 2.2.
No. 1 2 3 4 5 6 7 8 9 Action Select single object Select multiple objects Zoom-in Zoom-out Help Print Undo Redo Go home No. 10 11 12 13 14 15 16 17 18 Action Back Forward Rotate Open menu Close menu Move object Cut Copy Paste

Table 2.2 Set of actions to be used in the Guessability Testing

10

Guidelines for Gestural Tablet Interaction

2.4. Inter-Rater Agreement
2.4.1. Process
Recruit Particpants Planning Create Discussion Guide Inter-Rater Agreement Test Data Analysis Presentation of Data

Figure 2.4 Inter-Rater agreement process 2.4.2. Testing Method The inter-rater agreement method is similar to the categorisation tests that are utilised in icon testing, in which participants are asked to ‘arrange icons into meaningful groups’ to measure the ‘users’ recognition of the images and also their understanding of the concepts represented by the images’ (Horton 1994: 301). Following the guessability study, I had 18 gestures per action and I had to decipher a single gesture from all of those options that would be the sole representation for a specific action. For certain actions, many gestures were very similar, yet had individual traits that also made them slightly different. Therefore, I had to find a consensus, where multiple people could categorise the gestures, allowing me to recognise the similarities in participants’ groupings. These similarities would allow me to understand different participants’ mental models which will provide a solid foundation when crafting the set of guidelines for gestural tablet interaction. However, participants would have to categorise all of the gestures in one attempt, and with 18 gestures per action, with a total of 18 actions that would require the participants to view 324 gestures. Moreover, there was concern about participants forgetting why certain gestures were placed into certain categories as they progressed through all 324 gestures. Users would also probably review already-categorised gestures which would further prolong the test. Therefore, I decided to initially run a pilot test for this categorisation method so this method’s effectiveness could be evaluated. The pilot would involve 3 participants and individually, they were asked to group 90 gestures (based on 5 actions, each with 18 gestures), placing gestures that they felt were similar together. 2.4.3. Data Analysis I will be utilising the Fleiss’s Kappa method as it allows me to measure category agreement between several raters. The result from the Fleiss’s Kappa measure will be a number between 0 and 1, where a higher number would suggest more of an agreement between the raters, therefore suggesting a better agreed category. Based on this category score, I will be able to select the action/gesture pairings that are agreed upon by the participants of the inter-rater agreement study; resulting in a more reliable set of guidelines for gestural tablet interaction.

Chapter 2: Methodology

11

2.5. Gesture-Meaning Association Testing
2.5.1. Introduction The legitimacy of gestures hasn't been tested until recently. While there are many precedents in other aspects of usability study design, there is no common way to test a set of gestures to see whether the relation with the desired action is intuitive. Icons and gestures are, by definition, similar as they are signs or symbols that refer to a specific item or action. However, icons benefit from being a visual symbol that is placed on the visual user interface while gestures are invisible until the gesture is performed. 2.5.2. Process
Planning Recruit Particpants Gesture-Meaning Association Test Data Analysis Presentation of Data

Figure 2.5 Gesture-Meaning Association process 2.5.3. Participants When Jakob Nielsen conducted icon intuitiveness tests, a small set of users were invited to participate; around 3 - 5 users. While this is a small number, it was important to select the correct users as ‘no testing is effective without adequate test subjects.’ (Horton 1994: 293) and ensure that the test subjects reflect the tablets’ user base. Moreover, William Horton suggests that the participants shouldn't have ‘inappropriate knowledge of the product’ (1994: 293), however we can not ignore the fact that Apple - along with other manufacturers - are teaching people how to use the products through their multi-million dollar advertising campaigns and have to accept that this could affect some users’ decisions. 2.5.4. Testing Method There are two main methods to test the effectiveness of icons and they can be adapted to suit gesture evaluations too. William Horton suggests usability testing and image-meaning association analysis. When conducting usability testing, icons are placed 'in a prototype of the user interface and we test whether users can do meaningful work with the system.' This allows us to analyse how well the icons integrate 'with other aspects of user-interface design' (Horton 1994: 301). Even though this can also be done with lo-fidelity prototypes, it won't validate the set of gestures I create. To inform the design of the gestures I utilised a tablet device that participants used to create gestures based on a desired action. If this method was selected, we would merely be recreating the exact same test from the previous stage. Therefore, to validate the gesture set we need to reverse this concept and ask a different set of participants what they expect the gesture to do. For icon design, this is known as image-meaning association which measures 'the degree to which the graphic chosen for the icon represented the intended concept' (Nielsen 1995). For the purpose of gesture testing this method will be known as gesture-meaning association. As shown in the following chart, there are several options for gesture-meaning association. This project used the gesture-meaning matching method as it provided us with more data; allowing us to 12 Guidelines for Gestural Tablet Interaction

'observe which are matched first and which later' and 'which are matched quickly and which are matched only by a process of elimination' (Horton 1994: 300). This knowledge was fundamental in ranking the gestures in terms of user acceptance.
Method Open-ended meaning-forgesture Multiple-choice meaningfor-gesture Gesture-for-meaning Description Shows the participant the gesture and asks “What does this gesture mean?” This tests the association a user already has for a gesture. Presents the user with a gesture and a list of possible meanings for the user to pick from. This requires four or five plausible choices for the testing to be valid. Asks the user to identify which of several gestures best represents a concept. Can use it to test for completely different gestures or test different gestures for a single action. Gesture-meaning matching Provides participants with gestures and verbal labels for the same concepts but in a different order. Allows us to observe how quickly and accurately users match the image with the appropriate label.

Table 2.3 Gesture-meaning association options (Horton 1994: 297-300) Therefore, I had 32 pieces of paper; 18 slips that listed the actions and another 18 that showcased the gesture. The action slips were placed on one side while the gesture slips were placed on the other; random in order. I then, individually, asked the three participants to match the action/gesture pairs. 2.5.5. Data Analysis For the final aspect of this report, I analysed two performance measures as documented by William Horton (1994: 302):
Method Accuracy of selection Description Following the completion of the exercises, I checked the action/gesture relationships that were successfully paired and asked the participants about the pairings that were incorrect to discover why they felt their was a relationship between that specific action and gesture Speed with which selections are made I also noted the length of time it took for each pair to be chosen; paying special attention to the pairings that seemed to be created through a process of elimination

Table 2.4 Data analysis on the results from the Gesture-meaning association tests

Chapter 2: Methodology

13

2.6. Section Summary
2.6.1. General • There are three stages to this research; guessability study, inter-rater agreement testing and then gesture-meaning association testing. • These individual stages led on from each other and informed the proceeding stage. 2.6.2. Guessability Study • 18 participants individually created gestures for 18 commonly used computational actions. • 50% of the participants had prior tablet experience, while the other half hadn’t used a tablet device before. • The participants used an iPad to draw the gestures on, and a screenshot was taken for each gesture created while the entire test was video recorded. 2.6.3. Inter-Rater Agreement • It is a categorisation test that is used in icon usability testing. • Gestures for specific actions from the guessability study were similar, yet had unique traits that might differentiate them from one another to a variety of users. • Participants are asked to arrange the gestures created from the guessability study into categories. • Will allow me to understand users’ mental models. • The test would require each participant to watch all 324 gestures and categorise them. I was concerned over the possibility of participant fatigue. • Decided to run a pilot with 3 participants who would group 90 gestures based on 5 actions that each had 18 individual gestures. • If successful, the whole test will be initiated. 2.6.4. Gesture-Meaning Association Testing • Currently, there is no precedent in the process of validating gestures. • This test is also based on icon usability testing. • 32 slips of paper will be placed in front of a participant; 18 actions and 18 gestures. • The 3 participants will be asked to match a gesture to a specific action. • Will record accuracy of selection and speed with which selections are made.

14

Guidelines for Gestural Tablet Interaction

3. Formative Investigation
3.1. Introduction
In the previous section, I introduced the methodologies that would be utilised throughout this research; guessability study, inter-rater agreement and gesture-meaning association testing. Section 3 is an account on the first two stages; the guessability study and the inter-rater agreement. It will detail the issues discovered with the inter-rater agreement pilot that took place with three participants and how I revised that stage of the research to ensure that I would still produce similar ranking results to inform the final part of the project.

3.2. Guessability Study

Figure 4.1 Subject B participating in the Guessability Test As shown in Appendix A, I documented all 18 tests through consent and feedback forms, along with a monitoring sheet where I detailed my observations while the participants were creating their gestures. Appendix B showcases all 324 gestures that were created by the 18 participants. To ensure participants were relaxed and in an environment where they felt they would use a tablet device, I allowed the individual participant to decide where their test would take place. By using 18 separate locations, I had to continually setup the camera to record the iPad screen, ensuring the environment was well lit and that the device was clearly in shot. Yet, if I had utilised a single location for the tests, I would have saved a considerable amount of time as the test layout would have already been setup. While I believe the inconsistent locations provided more accurate results, the additional time did lead to increased participant stress due to the build-up process of setting up the test.

3.3. Inter-Rater Agreement
During the methodology section, I discussed the need to have a consensus on the type of gestures being performed - a tap, swipe, curve etc - and therefore decided to use the inter-rater agreement method. Yet there were concerns about the complexity and time-consuming nature of the method and as a result, I chose to run a pilot. For the pilot, I selected 3 participants to group 18 number of gestures from a set of 5 actions, with a total of 90 gestures overall. Individually, each participant was asked to watch a gesture video and Chapter 3: Formative Investigation 15

group the gestures on how they saw fit. This began successfully, however participants started to struggle as they reached the thirties and were constantly reviewing previous gesture videos to decide what group the current gesture belongs to. This was concerning because the study had amassed 324 gestures among 18 participants and 18 actions, yet the participants were struggling with only 10% of all gestures in the pilot. Therefore, it was decided that I should terminate the pilot and consider other options. 3.3.1. 2nd Stage Revision Instead, for each action I grouped the 18 gestures based on the type of gesture. Following this, I asked 3 people to rate the effectiveness of the gesture for that action; the same question that I asked the original guessability study participants. By averaging these ratings, I was able to have a consensus on whether specific gestures were a good match for the intended action. To stem the possibility of fatigue, participants were asked to rate only one action at any one sitting, amounting to only 18 gestures at a time. This process therefore, required a significant amount of time and lasted two weeks.

Figure 4.2 A rater participating in the replacement rating study By combining my grouping data along with their ratings I was able to extrapolate which type of gestures were accepted among participants. This data then allowed me to create the set of gestures.

16

Guidelines for Gestural Tablet Interaction

3.4. Section Summary
3.4.1. Guessability Study • This stage took place without any major issue. • Allowing the participant to decide the test location resulted in a time-consuming setup process for every test. This led to slightly increased participant stress. • 324 gestures were created by 18 participants. • Appendix A documents the consent, feedback and expert forms for all 18 participants. • Appendix B shows all 324 gestures that were created. 3.4.2. Inter-Rater Agreement • During the early stages, this stage was progressing well. • Raters struggled with the test when they reached around the 30th gesture of the test. • They began re-watching previous gesture videos to ensure they would put the current gesture in the correct group. • There are over 300 gestures, yet the raters were struggling with only 10% of the overall gesture amount. • Decided to terminate the pilot and select another method. 3.4.3. 2nd Stage Revision • I grouped the gestures based on the type of gesture - for example: tap, swipe, pinch. • 3 raters would rate each individual gesture on it’s relevancy to the specific action. • Asked the same question as the guessability study participants - ‘The gesture they picked is a good match for its intended purpose’ • Coupling the gesture groupings with the rating scores allows me to identify good matches for action/gesture relationships. • This stage will inform the design of the set of guidelines for gestural tablet interaction.

Chapter 3: Formative Investigation

17

4. Findings
4.1. Introduction
Chapter 3 discussed how the guessability study and inter-rater agreement pilot progressed. I also examined the issues concerning the inter-rater agreement pilot and why there was a need for another approach as the raters were struggling to complete the entire test. In this chapter, I will discuss the results from the first two stages; guessability study and the interrater agreement replacement test. The second stage was replaced with a basic rating routine that didn’t require the raters to go through the entire study in one sitting.

4.2. Guessability Study
Appendix B documents the type of gestures created by the participants for each action.

Figure 4.1 A selection of gestures for the Help action (top row) and the Close menu action 4.2.1. Desktop Metaphors Like other studies, I discovered that many participants were relying on the desktop-based metaphors that have been prominent for over 30 years. Even though I reiterated the multi-touch capability of the device - even demoing it to them prior to the user study - many users still only utilised a tap; akin to a single click or double click on a desktop mouse. This finding was also noted in the University of Washington study where ‘participants still often thought of the desktop paradigm’ (2009: 8), while my study found that 23% of all gestures were either single tap or double tap. This statistic is quite concerning as it could result in a reliance of desktop metaphors that were designed over 30 years ago and fail to take advantage of the unique multi-touch display that blankets the front of modern tablets. 4.2.2. The use of Letters and Symbols With several actions, many participants utilised letters or symbols as their choice of gestures; with in many cases being the most popular choice. This was a very interesting discovering as it was a huge shift from the desktop-based gestures that were more common for other actions in this study. Chapter 4: Findings 19

It is also very interesting to note how participants were drawing letters or words to describe the action they desired during the studies; letter or words associated with desktop-based shortcuts.

Tap and Hold

ʻPʼ Letter

Multi-Tap

Tap

Swipe 0 Strongly Disagree 2 4 Disagree 6 Neutral 8 Agree 10 Strongly Agree 12

Figure 4.2 Print action gestures For example, when asked what gesture they would choose to print a document, 6 out of 18 participants decided to draw the letter ‘P’ on the screen. Even though the letter ‘P’ is a rather common shortcut on the desktop - and would probably be used by inexperienced desktop users; one participant even utilised the more obscure shortcut letter ‘Y’ for redo. However, once participants thought of using a letter for an action, it seemed to be the default gesture for subsequent actions that were deemed challenging by the participants. This is in sharp contrast to the University of Washington study where they found the single tap gesture was utilised for difficult actions instead (Mauney et al 2010: 10). Moreover, I feel apprehensive in using symbols or letters as gestures due to their ‘potential for showing differences across cultures’ and therefore gestures will need to be adapted across cultures and countries (Mauney et al 2010: 6). Yet, I can’t dismiss the overwhelming support for symbolic gestures that were created by the participants of this study while developing my own gesture set. 4.2.3. Interaction with other objects When I asked Subject M what gesture they felt was suitable for printing, they performed a swipe gesture in the direction of the printer in their bedroom. Even though only one participant executed this gesture, I found it to be rather significant. When I asked him to explain their choice, Subject M stated that it felt natural to gesture in the direction of his printer, yet would not feel comfortable performing a similar move with a mouse on a desktop computer. Unfortunately, he was unable to further explain his choice of gesture, however, while observing the process, it seemed like he had forgotten about the constraints of desktop computing - one of the points I hoped all participants followed - and imagined how he would wish to interact with future technology.

20

Guidelines for Gestural Tablet Interaction

Aza Raskin believes that the job of an interface designer is to remove as much of the interface as possible so ‘you don’t even realize you are using’ an interface, because ‘the best interface is no interface' (Raskin 2008). It is therefore, these types of gestures that make ‘invisible interfaces’ a viable option in tablet devices - more so than on the desktop - due to the range of possible input methods. This finding is both a positive and negative result from my insistence of participants deciding on where the tests should take place - based on where they felt comfortable - as this result wouldn’t have been discovered in a closed lab environment, yet the lack of printers in other participant’s locations mean’t that this outcome was not possible for them. 4.2.4. Smartphone experience and advertising influence Many of the participants for this study were under 30 and were therefore aware of touchscreen devices; many having used a touch-based smartphone within the past couple of years. Therefore, even without having experience on a tablet device, many participants were aware of gestures utilised on the iPhone and Android-based smartphones; gestures that are still relevant for tablets. Moreover, many participants stated that the gestures they performed for certain actions, were based on the methods seen in television advertising for smartphones; especially the iPhone. While 50% of participants were required to have no tablet-based experience, the ubiquitous nature of touch-based smartphones influenced the results of this study. 4.2.5. Struggling to perform gestures Occasionally, some participants were unable to perform a gesture for a specific action. Before the guessability study, I presumed that some users would be unable to execute gestures for cut, copy and paste as I deemed them to be the most difficult actions in the set. However, it was interesting to discover that Participant C - an inexperienced, over 50s participant - was unable to perform both the zoom-in and zoom-out actions. The overwhelming majority of participants used the pinch gesture; a gesture that wasn’t performed by British users in a 2010 study (Mauney et al 2010) and therefore suggests that people can quickly adapt to different gestures for commonly used actions at this early stage in tablet computing. Yet, the inability of Participant C to create gestures for these actions and her concern for being unable to complete these tasks shows how the intimidating and inaccessible nature of desktop systems still remain for tablet devices, in regards to basic computer users who lack the experience of smartphone devices and aren’t influenced by the manufacturers’ adverts.

Chapter 4: Findings

21

4.2.6. Need for Context Sensitive Menus For many actions, participants requested context sensitive menus that would be displayed following a basic gesture. For the cut, copy and paste actions, 70% of participants asked for context sensitive menus following gestures such as: tap and hold, multi-tap, and tap as shown in Figure 4.3.

6% 13% 12% 6% 35% 29%

Tap and Hold

Multi-Tap

Tap

Letters & Symbols

Other

No Attempt

Figure 4.3 Percentage of gesture types for cut, copy & paste actions The IUP study also recognised similar results as their ‘participants consistently requested [context sensitive menus] for some actions, and they are particularly useful for novice users’ and therefore they created two gestures - for certain gestures - for both novice and expert users. While this study was trying to discover what gestures best suit certain actions, it is also important to uncover what actions are difficult to convey through a gesture and what ones require additional onscreen support to be completed.

4.3. Rating Study
4.3.1. Relationship between total number of similar gestures and user agreement Prior to the Rating Study, I expected that the gesture types that were performed the most would be rated with a higher score. While this was the case for the majority of actions, some saw the opposite. Tapping the top-right and swiping were the most popular gestures for the Close Menu action; accounting for over 60% of all created gestures. However, in the Rating Study, the Swipe gesture received a score of 2.9 approval and Top-Right Tapping had an approval rating of 3.7, yet it was the X gesture that received the highest approval score. Drawing an X on the menu received a score of 4.5, even though only one participant in the Guessability Study choose that gesture.

22

Guidelines for Gestural Tablet Interaction

6 5 4 3 2 1 0 Swipe ʻXʼ Letter Multi-Tap Pinch Tap Top-Right No Attempt Agreement Average Total No. of Gestures

Figure 4.4 Comparison between Agreement Averages and Total No. Of Gestures for Close Menu When creating the set of gestures, issues like these drastically affect what gestures I should select for a specific action. I can not create a gesture by combining two gestures as it will lead to a sign that was neither created by users nor approved by them. 4.3.2. Advertising Influence It was interesting to note the very high approval rating of 5 gestures across 5 actions; gestures that are widely used in advertising smartphones and tablet devices. These gestures received an approval rating of over 4.5 for their relevant action.
Action Select single object Zoom-in Zoom-out Back Forward Dominant Gesture Tap Reverse-pinch Pinch Swipe Swipe

Table 4.1 List of popular gestures among advertised gestures For example, the pinch and reverse-pinch gestures scored a 4.6 approval rating for the zoom-out and zoom-in actions respectively; compared to the 1.5 approval rating average of the other two gestures created by the Guessability Study participants. This finding was also noted by the IUP study and they questioned ‘whether the gestures performed were naturally instinctive’ due to the obvious influence from advertising. Moreover, it was these gesture/action combinations that were popular among both experienced and inexperienced participants; suggesting that people are able to learn and recall gestures that they have seen before; even without prior experience of performing the gesture (Mauney et al 2010: 12).

Chapter 4: Findings

23

4.3.3. Approval of Symbols & Letters as Gestures In section 4.2.2, I discussed my concern over some participants using letters as the default gesture when they reached a difficult action; therefore not challenging themselves to create other options. However, the Rating Study participants approved of the use of letters as gestures for certain actions as shown in Figure 4.5 which details the approval ratings of the Paste action gestures. 8

6

4

2

0 Tap and Hold ʻVʼ Letter ʻPʼ Letter Curve Multi-Tap Tap No Attempt Agreement Average Score Total No. of Gestures

Figure 4.5 High approval rating of the ‘V’ Letter As witnessed with this example, the ‘P’ letter received a low approval rating of 1.3. However, if we combine this information with data from the Print action, we discover that the Rating Study participants rate letters associated with the relevant desktop shortcuts higher than ones that are not. 8

6

4

2

0 Tap and Hold ʻPʼ Letter Multi-Tap Tap Swipe Figure 4.6 High approval rating of the ‘P’ Letter for the Print action Agreement Average Score 24 Guidelines for Gestural Tablet Interaction Total No. of Gestures

4.4. Section Summary
4.4.1. Guessability Study • There seemed to be a reliance on desktop-based metaphors as 23% of all gestures were either single-tap or double-tap. • The use of letters and symbols for gestures was very popular. Associated letters were mainly based on the desktop shortcut rather than the action word. Eg. V for the Paste action instead of P. • For the Print action, one participant swiped the text towards the physical location of his printer in their room. • Many participants used gestures they have learnt from smartphone advertising campaigns as these gestures are still relevant on tablet devices. • The request for context-sensitive menus was overwhelming. For the cut, copy and paste actions, 70% of participants asked for a context-sensitive menu. This suggests that not all actions can be easily defined by a single gesture. 4.4.2. Rating Study • Certain gestures that were more popular for specific actions didn’t necessarily score a good approval rating. • Sometimes gestures that were created only once would score a high approval rating. Eg. ‘X’ gesture for the Close menu action was implemented by one participant in the guessability study, yet scored an approval rating of 4.5 out of 5. • Widely advertised gestures scored an approval rating of 4.5 out of 5 or above by the raters. Eg. Pinch for the Zoom-out action and Swipe for the Back and Forward actions.

Chapter 4: Findings

25

5. The Gesture Set
5.1. Introduction
Chapter 4 explored the results from both the guessability study and the rating study, examining the most popular gestures created by participants and how popularity doesn’t necessarily improve a gesture’s approval rating among other users. I discussed the role of advertising and how it could have affected how instinctively the gestures were performed. Moreover, I discovered that using letters as gestures for certain actions were very popular, yet are not adopted by current tablet devices. Figure 5.1 details the guidelines for gestural tablet interaction that I created based on the findings from the two prior procedures; the guessability study and then the following rating study.

5.2. Guidelines for Gestural Tablet Interaction
Action Select single object Proposed Gesture Tap item

Select multiple objects

Tap and hold one item and sequentially tap other items

Chapter 5: The Gesture Set

27

Action Zoom-in

Proposed Gesture Reverse pinch with two fingers on the canvas

Zoom-out

Pinch with two fingers on the canvas

Help

Draw symbol ‘?’ anywhere on the canvas

?
Print Double-tap anywhere on the canvas to open a context sensitive menu Shortcut: Draw letter ‘P’ anywhere on the canvas
Context-Sensitive Menu

P

28

Guidelines for Gestural Tablet Interaction

Action Undo

Proposed Gesture Double-tap anywhere on the canvas to open a context sensitive menu Shortcut: Draw letter ‘Z’ anywhere on the canvas

Context-Sensitive Menu

Z
Double-tap anywhere on the canvas to open a context sensitive menu Shortcut: Draw letter ‘Y’ anywhere on the canvas

Redo

Context-Sensitive Menu

Y
Swipe down with one finger on the canvas to reach the bezel of the device

Go home

Back

Swipe with one finger from left to right on the canvas

Page 2

Page 1

Chapter 5: The Gesture Set

29

Action Forward

Proposed Gesture Swipe with one finger from right to left on the canvas

Page 1

Page 2

Rotate

Tap and hold the corner of the item and curve in the desired direction

Hello Hello

Hello

Hell

o

Open menu

Double-tap anywhere on the canvas to open the menu

Menu

Close menu

Draw letter ‘X’ inside of the menu box

\

Menu

X

Menu

30

Guidelines for Gestural Tablet Interaction

Action Move object

Proposed Gesture Tap item and then drag to the desired location

Cut

Double-tap anywhere on the canvas to open a context sensitive menu Shortcut: Draw letter ‘X’ on the item to be cut

Context-Sensitive Menu

X
Double-tap anywhere on the canvas to open a context sensitive menu Shortcut: Draw letter ‘C’ on the item to be copied

Copy

Context-Sensitive Menu

C
Double-tap anywhere on the canvas to open a context sensitive menu Shortcut: Draw letter ‘V’ on the location where the item is to be pasted

Paste

Context-Sensitive Menu

V
Chapter 5: The Gesture Set 31

5.2.1. Need for Shortcut options As 6 actions required a context-sensitive menu to be displayed following a gesture, there was a need to create shortcut gestures for these actions that would provide advanced users immediate access to that specific feature. The shortcut gestures were based on the letters used in desktop shortcuts as these are already well known and do not depend on cultural knowledge.

5.3. Section Summary
• Created a set of guidelines for gestural tablet interaction. • 18 gestures crafted for 18 actions. • Based on results and findings from the guessability study and rating study. • 6 actions have shortcut gestures associated with them as well because their primary gesture launches a context-sensitive menu.

32

Guidelines for Gestural Tablet Interaction

6. Summative Investigation
6.1. Introduction
The previous section introduced the set of guidelines for gestural tablet interaction that I had created based on the guessability study and rating test. I created a set of 18 gestures that were individually paired with a specific action; some functions requiring two gestures as the primary gesture only displayed a context-sensitive menu. As I have created the set of gestures, Chapter 6 details the findings and results from the validation stage which utilised the gesture-meaning association testing method. The results from this section will determine if the action/gesture relationships - and therefore the overall gesture set - are clear and incontestable.

6.2. Gesture-Meaning Association Testing
As shown in Appendix D, I conducted a Gesture-Meaning Association test with three participants where they paired 18 common computer actions to 18 gestures.

Figure 6.1 Raters during the Gesture-Meaning Association testing

Chapter 6: Summative Investigation

33

6.3. Alterations to Original Methodology
It was originally planned that all created gestures would be assessed through the Gesture-Meaning Association method, however not all the gestures created can be adequately assessed using this procedure. This concerns the gestures that require a context-sensitive menu to be displayed following a double-tap gesture and therefore these will not be assessed during this stage. However, all actions that utilised a context-sensitive menu, also had a shortcut gesture associated with it - for example: the letter ‘P’ for printing - and these will be assessed.

6.4. Results
The results from the gesture-meaning association test are shown in Table 6.1 and it details the number of participants who correctly matched the action/gesture relationship and the order in which a pairing was created.
Participant 1 Action/Gesture Pairings Select single object Select multiple objects Zoom-in Zoom-out Help Print Undo Redo Go home Back Forward Rotate Open menu Close menu Move object Cut Copy Paste Matched? ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✘ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Order 4 5 1 2 8 9 11 17 3 6 10 15 13 7 16 12 14 Participant 2 Matched? ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✘ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✘ ✔ ✔ Order 1 9 3 4 10 11 8 16 5 6 7 13 12 2 14 15 Participant 3 Matched? ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✘ ✔ ✔ ✔ ✘ ✔ ✔ ✔ ✔ ✔ Order 4 8 2 3 5 10 9 12 6 7 13 11 1 16 14 15

Table 6.1 Gesture-Meaning Association test results

34

Guidelines for Gestural Tablet Interaction

6.5. Findings
6.5.1. Gestures & Actions that were Easily Paired Out of the 18 actions, 9 of them - shown in Table 6.2 - were very easily paired with the correct gesture by all participants.
Easily Paired Actions Select single object Zoom-in Back Select multiple objects Zoom-out Forward Help Rotate Move object

Table 6.2 Easily paired actions While the majority of those gestures are already well known and accepted gestures - thanks to manufacturers’ advertising - I was pleased to see the Help action be easily paired. This ? gesture is currently not implemented by any major tablet OS - iOS, Android and WebOS - yet the universal symbol seems to be recognised and supported by all participants - across all three stages - in this report. 6.5.2. Difficult Actions & Gestures to Pair During this process, I discovered that there were three actions that all participants had difficulty in pairing with a gesture; Redo, Go home and Open menu. While the Redo and Open menu gestures are founded in desktop-based knowledge, the Go home gesture is rather distinct, even if it mimics the minimizing gesture on Windows and Mac OS X.

Figure 6.2 Minimizing effect on Mac OS X Therefore, the Go home action was one of the final actions to be paired with a gesture; often being chosen through a process of elimination. However, while discussing the action following the test, participants understood the reasoning behind the gesture, but two testers admitted that they are concerned by the hidden nature of the gesture. All of the shortcut gestures created as part of the set of gestures were based on the desktop shortcuts for that action, yet the ‘Y’ Redo shortcut is only known to Windows users as Mac OS X utilises another letter for the Redo function on their computers. This issue was only raised because two of the Gesture-Meaning Association participants were Mac OS X users and therefore were unable to associate the ‘Y’ gesture to the Redo action. 6.5.3. Unfounded Concerns over Cut, Copy and Paste Due to the trouble creating the gestures - both during the Guessability Study and the gesture production stages - for Cut, Copy and Paste, I predicted that these three actions would be difficult to pair up with the correct gestures. Yet, while these actions were some of the last to be paired Chapter 6: Summative Investigation 35

probably due to the simplicity of many other action/gesture pairings - two participants successfully paired these actions and gestures together. The participants were pleased that the gestures utilised their prior computing knowledge; echoing the support from a participant in the University of Washington study who stated ‘anything I can do that mimics Windows—makes my life easier’ (Wobbrock et al 2009: 8). 6.5.4. Confusion between Gestures & Actions As participants always had access to every action and gesture, they were able to change their decision at any time during the process. This allowed me to note any incident where gestures were paired with incorrect actions, only to be later paired with the correct action; informing me of possible confusion when the gestures are used as a group rather than as individuals. There was slight confusion with the ‘P’ gesture, as two participants originally paired the gesture with the Paste action instead of the correct Print action. This was before the participants realised that the shortcut gestures were based on their desktop counterparts and therefore the pairings were rectified before the end of the test.

36

Guidelines for Gestural Tablet Interaction

6.6. Section Summary
6.6.1. Alterations to Original Methodology • When creating the set of guidelines for gestural tablet interaction, 6 gestures had a primary gesture that displayed a context-sensitive menu. To assist advanced users, a shortcut gesture was included for those actions. • The primary gesture for the context-sensitive menu is difficult to validate because there is no direct connection between the action and the gesture. • The 6 gestures that displayed a context-sensitive menu were not validated, but the secondary shortcut gestures were validated using the gesture-meaning association test. 6.6.2. Findings • 9 action/gesture relationships were correctly paired very easily by all participants, 8 of which are actively used in manufacturers’ advertising campaigns. • The Redo, Go home and Open menu actions were very challenging for participants to correctly pair. The Redo shortcut is based on a Windows-only shortcut that isn’t available on Mac OS X. Other actions seemed to be paired by a process of elimination at the later stages of the gesturemeaning association test. • Cut, copy and paste actions were correctly paired by 2 participants. They were pleased they could utilise their desktop knowledge when using a tablet device. • Initial confusion between certain gestures and actions; two users initially placed the P gesture with the Print action, before correcting their mistake and placing it with the Paste action.

Chapter 6: Summative Investigation

37

7. Discussion & Conclusion
7.1. Conclusion
I have presented a study that examined and understood what gestures users associate with specific actions, leading to the creation of a set of guidelines for gestural tablet interaction. During the guessability study, I gathered 324 user-generated gestures across 18 actions, by 18 participants. These were then examined and grouped by gesture type, and given an approval score based on the appropriateness of the action/gesture relationship. Following this, I created a set of 18 action/gesture pairings which were then validated by another set of participants by utilising the gesture-meaning association method. It is hugely important that we can establish a set of common gestures shared across platforms and applications to eradicate the possibility of a fragmented interaction style on this new computing paradigm. If tablets follow a similar path as desktop computers, then they have the possibility of being in widespread usage for over 30 years. While desktop computers have improved their graphics and industrial design, the fundamentals of desktop computing hasn’t changed since their formation at Xerox PARC in the 1960s and 1970s. Therefore, it is pivotal that before we see widespread adoption of tablet devices, we must create a set of common gestures that can last a generation.

7.2. Value & Benefit
With several manufacturers gearing up to launch and release their tablet devices this year, I have created a set of platform-agnostic gestures, based on the input of 18 original participants who produced over 320 individual gestures across 18 actions. This report has shown that there is no need for platform-specific gestures and that a common set can be shared across a multitude of platforms. This therefore will allow users to move from platform to platform and utilise their habitual knowledge of a universal gesture set. Throughout this report, I emphasised the importance of user involvement as it allowed me to understand what gestures felt natural for them to perform, as well as understand participants’ level of action/gesture pairing acceptance. Users being the driving force behind the creation of the set of guidelines for gestural interaction should be an obvious choice, yet this approach is absent from current tablet implementations. Therefore, I involved current and potential tablet device owners, asking them to create, assess and validate the set of 18 gestures. Yet, this report discovered more than just 18 gestures, but provided information on action/gesture relationships that are not approved by users. This means we have the knowledge of what gestures to avoid in future so users don’t learn incorrect gestures and develop habit for bad practices. It is important to note that the set of gestures has been validated by another set of participants, which shows that these action/gesture relationships have been accepted and understood by a Chapter 7: Discussion & Conclusion 39

number of participants. Interestingly, the University of Washington study ignored this step and just released their set of gestures for surface computers based on a similar guessability study. While a gesture might be natural to draw, it doesn’t necessarily mean that the gesture can be interpreted as a depiction of a specific action.

7.3. Implications
Moreover, we witnessed gestures that are not included in current tablet devices; especially the use of letter or symbols. This discovery allows for the possibility of a huge number of potential gestures based on letters and symbols. It allows users to utilise their pre-existing computing and cultural knowledge on tablet devices; reducing the learning process of new users. However, it must be noted that while the majority of letter-based gestures created in this report are based on universallyaccepted desktop shortcuts, future letter-based gestures need to be tested for cultural differences. During the Guessability Study, we discovered that many participants were asking for contextsensitive menus to be displayed for certain actions; suggesting that not every function can be depicted as a gesture. Therefore, there needs to be a common gesture that will provide further actions and it could be argued that this gesture would be the most vital one as it will provide access to a number of actions.

7.4. Challenges
7.4.1. Involvement of Users My research was very user-driven, requiring them to participant throughout all three stages of the project. This was very time-consuming and challenging for both the participants and myself, yet without them, my report would have been worthless. Finding 18 participants for the guessability study that were both available and fit the set criteria was difficult as many potential participants were also completing academic projects. 7.4.2. Failed Pilot Study The failed inter-rater agreement pilot study also added additional pressure as it required me to research another method, design the test and then run it, thereby increasing the workload. This issue wasn’t anticipated at the start of the project; only arising at the beginning of February. While I overcame the issue - by finding raters eligible to participate within a week’s notice - it still brought unneeded pressure on both the report and my other academic responsibilities.

7.5. Future Work
7.5.1. Individuality of Gestures Even though we have created and validated individual gestures for this report, the gestures were not tested as an overall set. Like the International Usability Partners study, some participants created 'similar, or even the same, gestures … for dissimilar actions' (Mauney et al 2010: 13) and this therefore affected which gestures would be tied to a specific action - sometimes with the most popular one not being selected.

40

Guidelines for Gestural Tablet Interaction

Even though the study was conducted in real-world environments - based on where the participants felt comfortable and where they would consider using a tablet device - it remained platform and application agnostic in an attempt to limit the affect of pre-learned skill. However, there is a need to test the gesture set in real applications and with proper use cases to justify it's implementation in real system. Testing the gesture set in real-world applications has the opportunity of yielding different results to a study that was performed in isolation. 7.5.2. Different Types of Users Many different subset of users are having their first computer experiences on tablet devices. However, this study was based mainly on the limited age range of 20-25 - with a few exceptions and this ignores many other important age groups. Some old age pensioners have seen the potential of tablet devices that portray a friendlier and more user-friendly image to that of standard desktop computers (Wipf 2010), while the tactile nature of the devices are enticing toddlers to interact with them (Stewart 2010). Moreover, further research should discuss accessibility issues and involve disabled participants in creating and evaluating a gesture set. Even though basic applications have been created for certain disabled users, there is a need for an accessible gesture set that spans platforms rather than just individual applications (Apple 2011). 7.5.3. Further Analytical Work As I was unable to fully utilise the inter-rater agreement method due to the inability of the raters to complete the pilot task, the use of analytical data is very limited in this report. The replacement method only asked the raters to rate the gestures based on the appropriateness of the action/gesture relationship, and the three raters’ scores were then merged to create an average score. Future research should discuss and examine the need for increased analytical data and how it could assist in the creation of a more solid set of guidelines for gestural tablet interaction. 7.5.4. Reliance on Desktop Knowledge Further research should also discuss the impact of users’ reliance on desktop-based metaphors. To limit the affect of cultural differences, I based the shortcut gestures on the desktop-shortcut equivalent and not on the action word; for example: V for the Paste action instead of P. However, we need to examine whether this has any detrimental effects on tablet interaction and whether it limits the overall scope of interaction styles and techniques on tablet devices.

Chapter 7: Discussion & Conclusion

41

8. Project Management
8.1. Supervisor Meetings & Progress
The notes from the meetings with my supervisor can be found in Appendix F. Prior to the Interim Report, I had scheduled 4 meetings with my supervisor to discuss the progress that I had made on this final year project. We continued to have meetings throughout the academic year; roughly every 2 weeks. These meetings were pivotal in ensuring that my project remained on track as mini-deadlines were set up, so that my supervisor could see the project develop over time. Throughout the year, I attended all scheduled meetings with my supervisor. 8.1.1. 26th January Due to illness, I was unable to attend this meeting. However, I emailed my supervisor concerning the progress that I had made; informing him of the slight additions to the guessability study and that I was beginning the process of writing up the Methodology section. 8.1.2. 9th February We discussed the final two stages of the research; inter-rater agreement and the validation test. I raised concerns about the inter-rater agreement method as it would be rather time-consuming for the participants because it would take several hours for an individual to complete. Therefore, we decided to run a pilot test and John Halloran requested it to be completed by the next meeting. The literature review - the results of which can be found in the Introduction & Background section was completed prior to this meeting. As stated in my Interim Report, it was a challenge to find relevant literature as it is an emerging field. Therefore, I was limited to just two academic papers and one published book that discussed relevant gestural systems. Yet, the first section of this project benefits from the literature that discussed the forging of the desktop 30 years ago as many issues facing that generation still concerns the fledging tablet market. 8.1.3. 23rd February During this meeting, I brought back the results from the pilot study. As mentioned in Section 3, the pilot study was unsuccessful and therefore I had to select another approval rating method. As I was approaching the end of February, it was very concerning that I had to plan and complete another approval rating test and therefore we decided to take a slightly more simpler approach where raters would rate each gesture based on how well it represented the intended action. Moreover, I also completed the first draft for the Introduction & Background section, along with the Methodology section. My supervisor read through the draft during the meeting and informed me of the importance of referring back to the purpose of this research; the purpose has to permeate throughout the entire document.

Chapter 8: Project Management

43

8.1.4. 9th March The results of the new approval test were explored during this meeting. I informed my supervisor of the method utilised and how many participants were involved, as well as discussing how the results would be expressed in the project documentation. My supervisor also asked for a draft project by the next meeting in two weeks time. 8.1.5. 23rd March My supervisor reminded me of the four components that make up this module; the Interim Report, Presentation, Report, and Product Quality. As the Product Quality mark scheme wasn’t available online, my supervisor told me to request the mark scheme from Richard Rider. The supervisor also asked for a full draft of the report and that it be emailed to him for the 12th April. I emailed Richard Rider, and he uploaded the mark scheme online for all students. Even though I emailed my supervisor on the 11th April, he did not receive the report. Therefore, John Halloran contacted me on the 14th April asking why he was yet to receive the document. Unfortunately, I do not know why my supervisor didn’t receive the draft, yet we arranged to have a meeting on the 18th April. 8.1.6. 18th April For this meeting, I brought along the Results and Conclusion sections for my supervisor to review. While reading these sections, my supervisor was concerned that it was difficult to see progression from section to section and suggested that I had included a brief review and introduction at the start of each section, along with a summary at a section end. My supervisor asked me to implement these changes and send him a final draft over the next few days for him to review and reply with feedback. On the 20th April, I emailed him with my final draft and he stated that the report was progressing very well; only suggesting minor alterations. He reiterated the importance of binding and submitting the report as early as possible to avoid any issues during the final stages.

44

Guidelines for Gestural Tablet Interaction

References
Apple, 2011. iPad: Year One. [Online Video]. Available from: http://www.youtube.com/ watch?v=HpiVeC1Z3yI [accessed 10 April 2011] Apple, 2011. Apple Reports First Quarter Results. [Webpage]. Available from: http://www.apple.com/pr/ library/2011/01/18results.html [accessed 16 February 2011] Apple, 2007. How To Advert. [Video]. Cupertino, California : Apple. BBC News, 2010. Blackberry launches Playbook rival to iPad. [Webpage]. Available from: http://www.bbc.co.uk/news/ technology-11416190 [accessed 16 February 2011] Nielsen, J., 1995. Icon Usability. [Webpage]. Bloomberg, 2010. Apple Increases Tablet Share to 95 Percent as Android Slips. [Webpage]. Available from: http://www.bloomberg.com/ news/2010-11-02/apple-increases-tabletshare-to-95-percent-as-android-slips.html [accessed 16 February 2011] Ginsburg, S., 2010. Designing the iPhone User Experience: A User-Centered Approach to Sketching and Prototyping iPhone Apps. 1 ed. Upper Saddle River, NJ: Addison Wesley. Goodwin, K., 2009. Designing for the Digital Age: How to Create Human-Centered Products and Services. Indianapolis, IN: John Wiley & Sons. RIM, 2010. Research in Motion to Acquire QNX Software Unit from Harman International. [Webpage]. Available from: http://press.rim.com/ release.jsp?id=3766 [accessed 16 February 2011] References 45 Palm, 2011. HP webOS Think Beyond Event. [Online Video]. Available from: http://www.youtube.com/ watch?v=VnjwG7Z8AM8 [accessed 16 February 2011] Available from: http://www.useit.com/papers/ sun/icons.html [accessed 9 February 2011] Google, 2011. Android Event. [Online Video]. Available from: http://www.youtube.com/ watch?v=RfJuigJebRg [accessed 16 February 2011] Horton, W., 1994. The Icon Book: Visual Symbols for Computer Systems and Documentation (Wiley Technical Communication Library). 1 ed. New York: John Wiley & Sons. Mauney, D., Wirtanen, A., Capra, M., 2010. Cultural similarities and differences in userdefined gestures for touchscreen user interfaces. CHI EA '10 Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems.

Raskin, A., 2008. Don't Make Me Click. [Online Video]. Available from: http://www.youtube.com/ watch?v=EuELwq2ThJE [accessed 10 April 2011] Raskin, J., 2000. The Humane Interface: New Directions for Designing Interactive Systems. 1 ed. Reading, Mass.: Addison Wesley. Rubin, J., 1994. Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests (Wiley Technical Communications Library). New York: John Wiley & Sons. Saffer, D., 2008. Designing Gestural Interfaces: Touchscreens and Interactive Devices. 1 ed. California, United States: O'Reilly Media. Saffer, D., 2009. Designing for Interaction: Creating Innovative Applications and Devices (Voices That Matter). 2 ed. New Riders. Stewart, T., 2010. The iPad as a serious business tool for toddlers. [Webpage]. Available from: http://econsultancy.com/uk/ blog/6329-the-ipad-as-a-serious-businesstool-for-toddlers [accessed 10 April 2011] Wipf, A., 2010. Over-65 Audience. [Webpage]. Available from: http://culturalfuel.net/ 2010/04/20/ipad-a-push-for-silver-surfingtesting-the-ipad-with-a-65-target-audience/ [accessed 10 April 2011] Wobbrock, J. O., Ringel Morris, M., Wilson, A. D., 2009. User-Defined Gestures for Surface Computing. CHI '09 Proceedings of the 27th international conference on Human factors in 46 Guidelines for Gestural Tablet Interaction

computing systems. Yahoo News, 2010. iPad sales cross million twice as fast as original iPhone. [Webpage]. Available from: http://news.yahoo.com/s/ ytech_gadg/20100503/tc_ytech_gadg/ ytech_gadg_tc1901 [accessed 16 February 2011]

Bibliography
Benyon, D., Turner, P. & Turner, S., 2004. Designing Interactive Systems: People, Activities, Contexts, Technologies. 1 ed. Harlow, England: Addison Wesley. Cooper, A., Reimann, R. & Cronin, D., 2007. About Face 3: The Essentials of Interaction Design. 3Rev Ed ed. Indianapolis, IN: John Wiley & Sons. Kortum, P., 2008. HCI Beyond the GUI: Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces (Interactive Technologies). Amsterdam: Morgan Kaufmann. Moggridge, B., 2006. Designing Interactions. Cambridge, Mass.: MIT Press. Nielsen, J., Mack, R. L., 1994. Usability Inspection Methods. New York: John Wiley & Sons. Shneiderman, B., Plaisant, C., Cohen, M. & Jacobs, S., 2009. Designing the User Interface: International Version: Strategies for Effective Human-Computer Interaction: Strategies for Effective Human-Computer Interaction: International Edition. 5 ed. Pearson Education.

Bibliography

47

Appendix A
Guessability Study Documentation & Forms
Participant A

Appendix A

49

50

Guidelines for Gestural Tablet Interaction

Participant B

Appendix A

51

52

Guidelines for Gestural Tablet Interaction

Participant C

Appendix A

53

54

Guidelines for Gestural Tablet Interaction

Participant D

Appendix A

55

56

Guidelines for Gestural Tablet Interaction

Participant E

Appendix A

57

58

Guidelines for Gestural Tablet Interaction

Participant F

Appendix A

59

60

Guidelines for Gestural Tablet Interaction

Participant G

Appendix A

61

62

Guidelines for Gestural Tablet Interaction

Participant H

Appendix A

63

64

Guidelines for Gestural Tablet Interaction

Participant I

Appendix A

65

66

Guidelines for Gestural Tablet Interaction

Participant J

Appendix A

67

68

Guidelines for Gestural Tablet Interaction

Participant K

Appendix A

69

70

Guidelines for Gestural Tablet Interaction

Participant L

Appendix A

71

72

Guidelines for Gestural Tablet Interaction

Participant M

Appendix A

73

74

Guidelines for Gestural Tablet Interaction

Participant N

Appendix A

75

76

Guidelines for Gestural Tablet Interaction

Participant O

Appendix A

77

78

Guidelines for Gestural Tablet Interaction

Participant P

Appendix A

79

80

Guidelines for Gestural Tablet Interaction

Participant Q

Appendix A

81

82

Guidelines for Gestural Tablet Interaction

Participant R

Appendix A

83

84

Guidelines for Gestural Tablet Interaction

Appendix B
Gesture Screenshots
Participant A

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

85

Open menu

Close menu

Move object

Cut

Copy

Paste

86

Guidelines for Gestural Tablet Interaction

Participant B

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

87

Open menu

Close menu

Move object

Cut

Copy

Paste

88

Guidelines for Gestural Tablet Interaction

Participant C

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

89

Open menu

Close menu

Move object

Cut

Copy

Paste

90

Guidelines for Gestural Tablet Interaction

Participant D

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

91

Open menu

Close menu

Move object

Cut

Copy

Paste

92

Guidelines for Gestural Tablet Interaction

Participant D

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

93

Open menu

Close menu

Move object

Cut

Copy

Paste

94

Guidelines for Gestural Tablet Interaction

Participant E

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

95

Open menu

Close menu

Move object

Cut

Copy

Paste

96

Guidelines for Gestural Tablet Interaction

Participant F

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

97

Open menu

Close menu

Move object

Cut

Copy

Paste

98

Guidelines for Gestural Tablet Interaction

Participant G

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

99

Open menu

Close menu

Move object

Cut

Copy

Paste

100

Guidelines for Gestural Tablet Interaction

Participant H

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

101

Open menu

Close menu

Move object

Cut

Copy

Paste

102

Guidelines for Gestural Tablet Interaction

Participant I

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

103

Open menu

Close menu

Move object

Cut

Copy

Paste

104

Guidelines for Gestural Tablet Interaction

Participant J

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

105

Open menu

Close menu

Move object

Cut

Copy

Paste

106

Guidelines for Gestural Tablet Interaction

Participant K

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

107

Open menu

Close menu

Move object

Cut

Copy

Paste

108

Guidelines for Gestural Tablet Interaction

Participant L

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

109

Open menu

Close menu

Move object

Cut

Copy

Paste

110

Guidelines for Gestural Tablet Interaction

Participant M

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

111

Open menu

Close menu

Move object

Cut

Copy

Paste

112

Guidelines for Gestural Tablet Interaction

Participant N

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

113

Open menu

Close menu

Move object

Cut

Copy

Paste

114

Guidelines for Gestural Tablet Interaction

Participant O

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

115

Open menu

Close menu

Move object

Cut

Copy

Paste

116

Guidelines for Gestural Tablet Interaction

Participant P

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

117

Open menu

Close menu

Move object

Cut

Copy

Paste

118

Guidelines for Gestural Tablet Interaction

Participant Q

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

119

Open menu

Close menu

Move object

Cut

Copy

Paste

120

Guidelines for Gestural Tablet Interaction

Participant R

Select single object

Select multiple objects

Zoom-in

Zoom-out

Help

Print

Undo

Redo

Go home

Back

Forward

Rotate

Appendix B

121

Open menu

Close menu

Move object

Cut

Copy

Paste

122

Guidelines for Gestural Tablet Interaction

Appendix C
Rating Study Documentation & Forms

Appendix C

123

124

Guidelines for Gestural Tablet Interaction

Appendix C

125

Appendix D
Gesture Data Charts
Select single object

Tap

Multi-Tap

Circle

0 Strongly Disagree

4 Disagree

8 Neutral

12 Agree Strongly Agree

16

Select multiple objects Multi-Tap Consecutive

Multi-Finger Simultaneously

Number Items

Circle

Tap Consecutive 0 Strongly Disagree 2 Disagree 4 Neutral 6 Agree 8 10 12 Strongly Agree

Appendix D

127

Zoom-in

Finger Spread

Zoom-Bar Swipe

Multi-Tap

No Attempt

0 Strongly Disagree

2

4 Disagree

6 Neutral

8

10 Agree

12

14 16 Strongly Agree

Zoom-out

Finger Spread

Zoom-Bar Swipe

Tap

No Attempt

0 Strongly Disagree

2

4 Disagree

6 Neutral

8

10 Agree

12

14 16 Strongly Agree

128

Guidelines for Gestural Tablet Interaction

Help ʻ?ʼ Symbol

Tap

Multi-Tap

ʻHʼ Letter

No Attempt 0 2 Strongly Disagree 4 Disagree 6 Neutral 8 Agree 10 12 Strongly Agree

Print Tap and Hold

ʻPʼ Letter

Multi-Tap

Tap

Swipe 0 Strongly Disagree 2 Disagree 4 6 Neutral 8 Agree 10 12 Strongly Agree

Appendix D

129

Undo ʻZʼ Letter Swipe ʻ^ʼ Symbol Curve Multi-Tap Tap Tap and Hold Shake Device 0 Strongly Disagree 2 Disagree 4 6 Neutral 8 Agree 10 12 Strongly Agree

Redo Swipe Reverse ʻZʼ Letter ʻ^ʼ Symbol Curve Multi-Tap Tap Tap and Hold ʻYʼ Letter Shake Device 0 Strongly Disagree 2 Disagree 4 Neutral 6 8 Agree 10 12 Strongly Agree

130

Guidelines for Gestural Tablet Interaction

Go home Swipe Down ʻHʼ Letter Swipe Left Tap Top-Right ʻXʼ Letter 5 Finger Pinch Tap and Hold 0 Strongly Disagree 2 Disagree 4 6 Neutral 8 Agree 10 12 Strongly Agree

Back

Swipe Up

Swipe Left

Arrow

Tap Left

0 2 Strongly Disagree

4 Disagree

6 Neutral

8 Agree

10 12 Strongly Agree

Appendix D

131

Forward

Swipe Down

Swipe Right

Arrow

Tap Left

0 Strongly Disagree

2

4 Disagree

6 Neutral

8 Agree

10 12 Strongly Agree

Rotate

Multi-Finger Curve

Corner-Grab Curve

Single-Finger Curve

No Attempt

0 Strongly Disagree

2 Disagree

4 Neutral

6 Agree

8

10 12 Strongly Agree

132

Guidelines for Gestural Tablet Interaction

Open menu Multi-Tap Tap and Hold ʻMʼ Letter Swipe Tap No Attempt 0 Strongly Disagree 2 Disagree 4 6 Neutral 8 Agree 10 12 Strongly Agree

Close menu Swipe ʻXʼ Letter Multi-Tap Pinch Tap Top-Right No Attempt 0 Strongly Disagree 2 Disagree 4 6 Neutral 8 Agree 10 12 Strongly Agree

Appendix D

133

Move object

Swipe

Tap and Hold > Swipe

Arrow Swipe

Tap and Hold > Tap

0 Strongly Disagree

2 4 Disagree

6 Neutral

8

10 Agree

12 14 16 Strongly Agree

Cut Tap and Hold ʻXʼ Letter ʻCʼ Letter Swipe Scrub Multi-Tap Tap No Attempt 0 Strongly Disagree 2 Disagree 4 6 Neutral 8 Agree 10 12 Strongly Agree

134

Guidelines for Gestural Tablet Interaction

Copy Tap and Hold ʻCʼ Letter Swipe Curve Multi-Tap Tap No Attempt 0 Strongly Disagree 2 Disagree 4 6 Neutral 8 Agree 10 12 Strongly Agree

Paste Tap and Hold ʻVʼ Letter ʻPʼ Letter Curve Multi-Tap Tap No Attempt 0 Strongly Disagree 2 Disagree 4 6 Neutral 8 Agree 10 12 Strongly Agree

Appendix D

135

Appendix E
Gesture-Meaning Association Results
Participant 1 Action/Gesture Pairings Select single object Select multiple objects Zoom-in Zoom-out Help Print Undo Redo Go home Back Forward Rotate Open menu Close menu Move object Cut Copy Paste Matched? ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✘ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Order 4 5 1 2 8 9 11 17 3 6 10 15 13 7 16 12 14 Participant 2 Matched? ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✘ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✘ ✔ ✔ Order 1 9 3 4 10 11 8 16 5 6 7 13 12 2 14 15 Participant 3 Matched? ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✘ ✔ ✔ ✔ ✘ ✔ ✔ ✔ ✔ ✔ Order 4 8 2 3 5 10 9 12 6 7 13 11 1 16 14 15

Appendix E

137

Appendix F
Supervisor Correspondence
25th January - Thomas Davies to John Halloran // Subject: 26th January Attendance John, I'm sorry but I am not feeling too well and therefore I don't think I will be able to make it to tomorrow's meeting. This week I have started the guessability study and have completed four tests at the moment. As I wasn't too sure about just having the before and after screenshot, I am showing participants an animation of the action (Eg. Moving an object action will show the object moving across the screen) prior to them creating the gesture. The gesture is still drawn on top of the before screenshot though. The entire process is being recorded so I can ensure all gestures and the order in which they occur are documented. I am also observing the participants and having them think aloud. I have also nearly completed the methodology section of the write up. Regards, Thomas

Appendix E

139

9th February - John Halloran to Thomas Davies // Subject: 9th February Meeting Notes IMAGE DEFINITION inter-rater agreement get a panel of people working independently show them video of gestures ask them to create clusters where they think the gesture is the same get THEM to come up with clusters; diff raters' clusters should agree ask raters for confidence levels when they put an item in cluster (5 point Likert scale) they can have a 'misc' pile cronbach's alpha inter rater reliability (google)

PILOT STUDY 10 gestures 5 users (raters would need to have some experience of this type of computing)

VALIDATION image-meaning association - validation aim for 2 weeks on the pilot

--------------------------------------------------Dr John Halloran BSc, MSc, PhD Department of Computing and the Digital Environment Coventry University Coventry CV1 5FB

john DOT halloran AT coventry DOT ac DOT uk

ext 7631

140

Guidelines for Gestural Tablet Interaction

23rd February - John Halloran to Thomas Davies // Subject: 23rd February Meeting Notes change study pairwise rater agreements within categories 18 categories 2x9 maximum of 9 users (say) - 2 each

- stats. difficult! needs SPSS; and what is the right stat. - I'll do a bit of research - looks like fliess of cohen (not cronbach) - work out the logic of the thesis

- we are looking for a standard set - intuitive and recognisable - so what? - supports the thesis that physical knowledge is basic, universal, not learned so easily deployed for this interaction style and validates interaction style (lit about people who say it is 'natural' / innate vs those who say rubbish it is all learned)  - support developers - supports users - where manufacturers deviate it is going to be harder to learn - independent user test of the results

j

--------------------------------------------------Dr John Halloran BSc, MSc, PhD Department of Computing and the Digital Environment Coventry University Coventry CV1 5FB

john DOT halloran AT coventry DOT ac DOT uk

ext 7631 Appendix E 141

9th March - John Halloran to Thomas Davies // Subject: 9th March Meeting Notes 'Eyeball' shows inter-rater agreement within categories 'do you believe this gesture suits the desired action' - Likert 1-5 3 raters

we need a chart, as I scribbled need a test for statistically significant agreement between raters I do not know what this test is!

Next time 23rd March - I want to see a draft dissertation - if not done, put placeholder content e.g. 'What is going to appear here is as follows' - Get the logic of the thesis and how the studies support it - I am away 4th-11th inclusive (but not otherwise) - aim at a longer supervision on 30th

1. we show that there is statistically significant agreement between raters

2. implement the gesture set     - description / storyboard

3. validate it     - 'image meaning test' --------------------------------------------------Dr John Halloran BSc, MSc, PhD Department of Computing and the Digital Environment Coventry University Coventry CV1 5FB

john DOT halloran AT coventry DOT ac DOT uk

ext 7631 142 Guidelines for Gestural Tablet Interaction

23rd March - John Halloran to Thomas Davies // Subject: 23rd March Meeting Notes Make sure there is a full draft dissertation on my desktop 9am Tues 12th April for comment

Keep hammering the research question, what it is, where we are with it, what got established so far and what's next and why it's relevant

Then we will arrange a meet in week 18-21 if need be

Check the project guide esp the marking criteria

remember there are FOUR components - brief (10), poster (10), report (40), 'product/quality' (40)

what are the forms? request from Richard Rider

--------------------------------------------------Dr John Halloran BSc, MSc, PhD Department of Computing and the Digital Environment Coventry University Coventry CV1 5FB

john DOT halloran AT coventry DOT ac DOT uk

ext 7631

| DREAMSEQUENCE | THE ANNUAL CREATIVE COMPUTING SHOW | Friday 6 May 16:00, Humber Theatre, Coventry University | www.dreamsequence.co.uk |

Appendix E

143

14th April - John Halloran to Thomas Davies // Subject: Project Tom I have had nothing from you, which is uncharacteristic. I am guessing this could be server rejection but am a bit worried it might be illness for e.g. Let me know what's happening. We'll probably need to meet next week Regards John 15th April - Thomas Davies to John Halloran // RE Subject: Project John, Thanks for emailing me. I emailed it in the early hours of Tuesday but it obviously didn't go through.  I'm not at home right now so I won't be able to resend it until the evening. I understand that it will be difficult for you to read through it now though so don't worry.  In that email, I also asked if it would be possible for you to act as one of my references for my application to City London. There are no immediate time constraints for the reference though. I've asked Peter to be my second reference at the same time - he got my email!! Thank, Thomas 15th April - John Halloran to Thomas Davies // RE RE Subject: Project OK. Yes happy to be a ref What about meeting next week? Say Mon; I could do 15:00 or 12:00 Regards John 15th April - Thomas Davies to John Halloran // RE RE RE Subject: Project 12:00 will be great.  Thanks, Thomas 15th April - John Halloran to Thomas Davies // RE RE RE RE Subject: Project OK confirmed See you then Regards John

144

Guidelines for Gestural Tablet Interaction

20th April - Thomas Davies to John Halloran // Dissertation John, Here is the link to my dissertation - http://files.me.com/thomasdavies/e9jcls Also, as you didn't receive my original email, here is also the reference document for the City University post-grad course - http://files.me.com/thomasdavies/eautlv Regards, Thomas 20th April - John Halloran to Thomas Davies // RE Dissertation just confirming I've received this - I can download the doic no problem expect feedback by close of play tomorrow Regards John 20th April - John Halloran to Thomas Davies // RE RE Dissertation Thomas This is looking excellent. The only issue is the contents page doesn't match the sections - in particular I can't see a project management section, and the page nos look wrong. Make sure the project management section appears and that the sections in general are consistent with the contents page. I trust you to do this without further guidance. I suggest you try to get it printed and bound before next week and submit asap which I guess will be Tuesday. I am confident that you know what the procedure is. Good Luck Regards John

Appendix E

145

Sign up to vote on this title
UsefulNot useful