Journal Pre-proof

Influence of Cultural Factors on Freehand Gesture Design

Huiyue Wu , Jinxuan Gai , Yu Wang , Jiayi Liu , Jiali Qiu ,

Jianmin Wang , Xiaolong (Luke) Zhang

PII: S1071-5819(20)30104-X
DOI: https://doi.org/10.1016/j.ijhcs.2020.102502
Reference: YIJHC 102502

To appear in: International Journal of Human-Computer Studies

Received date: 20 February 2019

Revised date: 11 June 2020
Accepted date: 26 June 2020

Please cite this article as: Huiyue Wu , Jinxuan Gai , Yu Wang , Jiayi Liu , Jiali Qiu ,
Jianmin Wang , Xiaolong (Luke) Zhang , Influence of Cultural Factors on Free-
hand Gesture Design, International Journal of Human-Computer Studies (2020), doi:
https://doi.org/10.1016/j.ijhcs.2020.102502

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition
of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of
record. This version will undergo additional copyediting, typesetting and review before it is published
in its final form, but we are providing this version to give early visibility of the article. Please note that,
during the production process, errors may be discovered which could affect the content, and all legal
disclaimers that apply to the journal pertain.

© 2020 Published by Elsevier Ltd.

Highlights
 We provided empirical evidence showing the existence of cultural bias on freehand gesture interaction, of which
designers should be aware in gesture design;
 We discovered cultural similarities and differences in the use of freehand gestures in different application domains,
among which 34 gestures are less influenced by cultural factors while 13 gestures are significantly influenced by
participants’ cultural background;
 We identified the types of tasks that exhibit significant difference between cultures, as well as the types of tasks that are
less sensitive to cultural factors; and
 We derived some design guidelines for freehand gesture design, which we hope can also be informative to other related
gesture-based design.

1
Influence of Cultural Factors on Freehand
Gesture Design
Huiyue Wu1,2, Jinxuan Gai3, Yu Wang4, Jiayi Liu1, Jiali Qiu1, Jianmin Wang5, Xiaolong
(Luke) Zhang3
1
The School of Communication and Design, Sun Yat-sen University, Guangzhou, China
2
Guangdong Key Laboratory for Big Data Analysis and Simulation of Public Opinion, Guangzhou, China
3
College of Information Sciences and Technology, Pennsylvania State University, University Park, PA 16802, USA
4
Department of Medical Oncology, Sun Yat-sen University Cancer Center; State Key Laboratory of Oncology, South China; Collaborative
Innovation Center for Cancer Medicine, Guangzhou, China.
5
User Experience Lab, College of Arts and Media, Tongji University, Shanghai, China

This work was supported by the National Natural Science Foundation of China (61772564, 61772468) and the National Key Research and Development
Program of China (2018YFB1004903)
Contact Huiyue Wu: wuhuiyue@mail.sysu.edu.cn

ABSTRACT. In the design of gesture-based user interfaces, gesture elicitation methods are often used to
obtain gesture preferences of end users. User choices of gestures can be affected by various factors, such as
their cultural backgrounds. Considering cultural factors in designing gesture-based interfaces is important to
systems that target for users from different cultures. However, so little empirical research has been
conducted on the impact of cultures on gesture commands. This paper reports a study with a series of three
experiments on the gesture preferences for tasks in three different application domains with participants
from two cultures. We find that some gesture choices are strongly influenced by the cultural background of
participants. We discuss the characteristics of those gestures that exhibit cultural dependence and those
tasks that do not. We also provide some design guidelines for freehand gesture-based interfaces.

INDEX TERMS—Gestural interaction, user-defined gestures, elicitation study, cultural bias

1 Introduction commands, on the contrary, have been the focus of

Gesture-based interfaces in interactive systems become more
and more popular (Han et al., 2013; Turk, 2014; Rautaray et
al., 2015; Cheng et al., 2016; Cai et al., 2017). Gesture-based
interaction designs let people use the body movement in the
physical space in direct control of information space,
providing end-users with more interaction freedoms, larger
interaction space, and more life-like interactive experiences.
Therefore, gestures-based commands, such as freehand
gestures, are widely applied in various application domains,
such as virtual reality (VR) and augmented reality (AR)
(Feng et al., 2013; Piumsomboon et al., 2013), computer
games (Kulshreshth et al., 2014), wearable computing (Tung
et al., 2015; Shimon et al., 2016; Gheran et al., 2018), smart
homes (Takahashi et al., 2013; Wu et al., 2016), robots
(Yang et al., 2007; Obaid et al., 2012) and UAVs (Pfeil et al.,
2013; Peshkova et al., 2017).
End-users are often involved in the design of gesture
commands. With the rapid development of technologies in
the areas of vision-based sensors, biocybernetics, markerless
motion capturing, and natural Human-Computer Interaction
(HCI), the accuracy of gesture recognition has become less a
concern. The usability and social acceptability of gesture
researchers and designers. One of the commonly used
approaches is to let end-users participate in design processes
(Wobbrock et al., 2009). However, for end-users, what
gestures to choose for a specific system may be influenced
by various factors, such as personal preferences and their
cultural backgrounds. It has been reported (Brown et al.,
2008) that in nonverbal communication, the frequency,
rhythm, viewpoint and description of motion events are
clearly culture-related. Such findings imply that gestures
based on freehand movement may exhibit culture-related
characteristics. To design better gesture-based user interfaces,
researchers and designers need a good understanding of how
cultures may influence freehand gestures.
Although some research has investigated the relationship
between gestures and cultures, such as Mauney et al. (2010),
a study on touchscreen gesture choices by users from
different cultures, little research has been done to examine
the influences of cultures on freehand gestures, which
without the constraint of physical interactive artifacts (e.g.,
data glove, touch screen), give users more freedom in gesture
choices. In this paper, we report a study on the differences in
gesture choices between users from two countries (USA and

2
China) in three different application domains: interacting
with equipment in a car, interacting with a virtual reality
environment, and interacting with a TV. Our results identify
some tasks and task categories that could be significantly
influenced by culture. Our research makes two major
contributions. First, we provide empirical evidence on the
existence of the cultural bias problem in freehand gesture
design. Second, our research offers insight into the cultural
similarities and differences in user-defined gestures for these
application domains. We identify some task types that are
influenced by the cultural background of users, as well as
some task types that are more consistent across cultures.We
also provided design guidelines for freehand gesture design.
We believe that these findings will benefit research and
design practices of freehand gesture user interaction.
The remainder of this paper is structured as follows: First,
we review related work and introduce our research
methodology. Then, we describe our three-stage
experimental studies, report their results, and discuss our
findings. Finally, we conclude the paper by discussing the
contributions of our research and possible future research
directions.
2 Related work
Freehand gestures have been used in various ways. Some
applications simply used gestures as a “natural mouse” for
such tasks as pointing and drawing (Kristensson et al.,
2012; Tian et al., 2017), while others used gestures for
more complex activities, such as manipulating virtual
objects (Song et al., 2012; Feng et al., 2013; Alkemade et
al., 2017), interacting with large displays (Hilliges et al.,
2009; Rovelo et al., 2014), and controlling smart home
devices (Takahashi et al., 2013; Wang et al., 2015; Zaiţi et
al., 2015).
However, traditional freehand-gesture-based systems are
often designed without fully consulting end-users, or
sacrificing usability for ease of implementation and
practical reasons (Chan et al., 2016). Such practices may
lead to a disagreement between user gestures imagined by
system designers and actual user gestures. Similar to the
vocabulary problem (Furnas et al., 1987) that affects the
performance of information retrieval systems, this gesture
disagreement problem (Wu et al., 2018) may cause poor
system usability and low user acceptance.
Gesture elicitation studies, a technique emerging from
participatory design, have attracted increasing attention and
been widely used to collect end-users’ requirements of and
expectations for a target system by involving end-users in
gesture design processes. Morris et al. (2010), for example,
provided empirical evidence on the benefits of involving
end-users in gesture elicitation studies. Their results show
that gestures authored by end users are easier to memorize
and discover than those created by researchers or designers.
Recently, gesture elicitation studies have been widely
applied to such emerging HCI devices and applications as
surface computing (Wobbrock et al., 2005; Wobbrock et al.,
3
2009; Kurdyukova et al., 2012; Buchanan et al., 2013;
Grijincu et al., 2014; Valdes et al, 2014), mobile interaction
(Kray et al., 2010; Ruiz et al., 2011; Seyed et al., 2012;
Chan et al., 2016), virtual/augmented reality
(Piumsomboon et al., 2013; Connell et al., 2013; Lee et al.,
2015), large displays (Morris 2012; Nebeling et al., 2014;
Rovelo et al., 2014; Lou et al., 2018; Chen et al., 2018),
wearable devices (Tung et al., 2015; Shimon et al., 2016;
Gheran et al., 2018), in-vehicle information systems
(Döring et al., 2011; Angelini et al., 2014), a humanoid
robot (Obaid et al., 2012), and smart-home applications
(Kühnel et al., 2011; Löcken et al., 2012; Vatavu, 2012;
Zaiţi et al., 2015; Dong et al., 2015; Wu et al., 2016).
However, traditional gesture elicitation studies often
suffer from the legacy bias problem (Morris et al., 2014) in
practice, which refers to the phenomenon that without any
constraint in gesture design process, participants’ proposals
are often biased by their experience with prior interfaces
and technologies such as the WIMP interfaces (e.g., a
mouse) or touch-based interfaces (e.g., an iPhone). As a
result, the choices of gestures by end-users could be very
diverse. According to existing literature (Kühnel et al.
2011; Vatavu, 2012; Wu et al., 2016), legacy bias may have
a significant effect on results in a gesture elicitation study.
Recently, the legacy bias problem has attracted increasing
attention and been investigated in many freehand-gesture-
based applications (Dong et al., 2015; Chan et al., 2016;
Hoff et al., 2016; Gheran et al., 2018; Wu et al., 2018;
Chen et al., 2018).
In addition to experience with prior interfaces and
technologies, other factors, such as the cultural
backgrounds of end users, may also lead to biases and
produce different gestures for the same target tasks. For
example, for the same Turn off the TV command in a TV-
based application, the user-defined gestures developed by
end-users from different cultural groups vary significantly,
as shown by Vatavu et al. (2012), Kühnel et al. (2011), and
Wu et al. (2016). Such culture-related differences in
designed gestures may lead to more challenges in technical
design (e.g., the recognition algorithms). Although early
research has hinted the existence of this problem (Rehm et
al., 2008; Mauney et al., 2010; Ruiz et al., 2011; Vatavu et
al., 2012; Nacenta et al., 2013; Morgado et al., 2013;
Rovelo et al., 2014; Serrano et al., 2014; Tung et al., 2015;
Vatavu et al., 2016; Jane et al., 2017; Chen et al., 2018), no
research has systematically investigated this issue. Our
research is an effort to examine this problem by conducting
experiments to compare freehand gestures designed by
users from two different cultural backgrounds in three
different application domains.
3 Methodology
Our research includes several steps. First, we selected three
application domains and identified some typical tasks that
are controlled by freehand gestures. Then, based on these
tasks, we conducted a series of experiments to investigate
the influence of cultures on designed gestures. In this task for easy reference in the rest of the paper. For example,
section, we describe these steps in detail. task 4 from the Car scenario is identified as “C4. Next
track” and task 5 from the VR scenario as “V5. Stop”.
3.1 Application Domains and Tasks
To make our study more general, we chose three 3.2 Experiment Studies
application domains that represent different use contexts of Based on the 70 target tasks, we designed a three-stage
freehand gestures. Three application domains are study. At the first stage, two different cultural groups (24
interacting with virtual reality (VR), which represents a American participants vs. 24 Chinese participants) were
typical environment in a lab or a classroom, commanding invited to a standard elicitation study to design gestures for
an interactive TV (TV), which represents a typical the 70 tasks. The aim of this experiment is to examine the
application scenario at home, and controlling non-driving gesture designs from these two groups when no restriction
tasks (e.g., navigation and entertainment) inside a car (i.e., a memory recall procedure) is imposed. At the second
(Car), which represents a non-traditional interactive stage, two new groups (still 24 vs. 24) were asked to choose
environment (e.g., using one hand rather than two hands for a favorite gesture for each of the 70 target tasks among
gestures). some gestures derived from the first stage. The goal of the
With these three application domains, we organized a second stage is to compare the difference in gesture design
brainstorm session to discuss the most-needed target tasks between two cultural groups when design choices are
for these three application domains. Fifteen HCI provided. At the third stage, with two new groups (24 vs.
researchers and graduate students were invited to the 24), we conducted an experiment to examine their
brainstorm session. Its outcome is 70 tasks (Table 1) that difference in designing gestures for 32 tasks that were
participants agreed important tasks for these application affected by culture significantly based on the results of the
domains. previous two experiments. The purpose of this experiment
is that in addition to further validating the findings of two
Table 1 experiments, we hope to identify what kinds of gestures
70 target tasks in 3 application domains could be influenced by cultural factors.
T# Target tasks T# Target tasks
C1 Play CD V9 Close the door
C2 Stop CD V10 Open the box 3.2.1 Experiment 1
C3 Previous track V11 Pick up an object In this experiment, we aimed to identify the cultural bias
C4 Next track V12 Close the box problem in different application domains and among
C5 Volume up V13 Confirm
C6 Volume down V14 Cancel
participants with different cultural backgrounds.
C7 Pause music V15 Single selection
C8 Resume music V16 Release object Participants
C9 Turn on radio V17 Group selection
C10 Turn off radio V18 Multiple selection
In this study, we recruited 24 participants (17 males and 7
C11 Next station V19 Move object females) from a university in the US and 24 participants
C12 Previous station V20 Rotate object (11 males and 13 females) from a university in China.
C13 CD to radio V21 Enlarge object Participants came from different majors and professional
C14 Radio to CD V22 Shrink object
C15 Pick up the phone V23 Copy object
backgrounds and their ages were between 18 and 28. We
C16 Switch to the previous call V24 Delete object chose these participants because of two reasons: 1) our
C17 Hang up the phone T1 Turn on the TV limit access to subject pools beyond this age group within a
C18 Reject a call T2 Turn off the TV university, and b) our expectation for participants who can
C19 Turn on air conditioner T3 Cancel accept and learn new technologies with little difficulty.
C20 Turn off air conditioner T4 Confirm
C21 Temperature up T5 Next channel
None of these 48 participants had any experience with
C22 Temperature down T6 Previous channel gesture design and freehand-gesture-based interaction
C23 Fan speed up T7 Volume up techniques (e.g., Kinect and Leap Motion).
C24 Fan speed down T8 Volume down
C25 Zoom in the map T9 Mute
C26 Zoom out the map T10 Cancel mute Apparatus
C27 Move the map T11 Change to channel 168 This study was conducted in two usability labs
V1 Move forward T12 Change to channel 79 simultaneously, one in the US and the other in China. Figure
V2 Step back T13 Change to channel 3
V3 Accelerate T14 Call channel list
1 shows the experimental environments of the three
V4 Slow down T15 Scroll down channel list scenarios.
V5 Stop T16 Scroll up channel list
V6 Turn left T17 Play a selected channel
V7 Turn right T18 Call main menu
V8 Open the door T19 Return to main menu

Table 1 lists the 70 target tasks collected from 3 different

application domains. We created a nomenclature for each
4

(a) the Car scenario
(b) the VR scenario
(c) the TV scenario
Figure 1. Experimental setup of the three scenarios
For the Car scenario, participants were asked to sit in the
driver’s seat of a real car as shown in Figure 1 (a). During
the experiment, participants were given the verbal
instructions of the 27 target tasks (Table 1) from an
experimenter sitting on the passenger seat in the back of the
car. As soon as participants heard the instruction of a target
task, they were required to perform a favorite gesture.
For the VR scenario, we provided an immersive VR
environment that incorporated HTC VIVE and the Leap
Motion controller as shown in Figure 1 (b). During the
5
experiment, the 24 target tasks (Table 1) were presented to
participants in forms of 3D virtual objects, textual
descriptions, video clips, and 3D animations through the
HTC VIVE. In this scenario, the Leap Motion controller
fixed on the top of the HTC VIVE was used to capture and
record participants’ hand postures and movements.
For the TV scenario, participants were asked to sit in a
sofa positioned 6 feet away from a 55-inch interactive
digital TV. A Microsoft’s Kinect sensor mounted on the top
of the TV was used to capture participants’ gesture
behaviors. Nineteen target tasks (Table 1) were shown to
participants in forms of textural description and 2D
animations on the TV screen.
For the purpose of consistency, participants in both
cultural groups among all three scenarios were exposed to
the same experimental interfaces, i.e., the car, HTC VIVE
HMD, and TV with the same make and model. During the
experiment, we also used a web camera accompanied with
a voice recorder to capture what participants did and said in
each scenario. To avoid bias, participants were not allowed
to given any hints from our experimenter.
Procedure
During the experiment, participants were first briefly
informed the experimental objective and then went through
the consent process. For the Car scenario, participants were
asked to performing hand gestures using their right hand
while keeping their left hand on the steering wheel. While
for the VR and TV scenarios, they can use both hands to
perform gestures. When performing a gesture, they were
required to verbally explain the reason they chose this
gesture for a given target task.
A typical experimental session lasted about one and a
half hours. After the experiment, participants were asked to
answer a short questionnaire on their demographic data,
including age, gender, profession, and their opinions on
freehand-gesture-based interaction techniques.
Results
Data analysis and processing
With 2 cultural groups, 24 participants in each group, and
70 target tasks, we collected a total of 2360 gestures (2 ×
24 × 70). With the data, we did a brainstorm session in
which 5 researchers with expertise in interaction design and
freehand-gesture-based interaction techniques grouped and
merged gestures for each task. We determined the gesture
set with the following rules:
 For those gestures with the exact same posture and/or
same movement trajectories, we grouped them into a
single gesture directly.
 For those gestures with similar but slightly different
characteristics in shape and/or trajectory, we replayed
the corresponding video/audio files captured by the
camera and recorder and then discuss whether and
how to group them based on participants’ verbal
explanations on those gestures.
 For example, twenty-two swiping-up actions with 2
|P |  | Pi |  1 (1)
different hand shapes can actually be merged into one A R (r )   
| P |  1 Pi  P  | P | 
 
| P | 1
group of identical gestures, as participants stated that they
preferred to mimic interactions in real-world scenarios where P is the set of all proposed gestures for task r, |P| the
during the gesture elicitation study: size of the set, and Pi subsets of identical gestures from P.
Based on the above formula, we calculated the
“I would use a Swipe Up gesture for C5 – Volume Up in differences in agreement between the American group and
the Car scenario. It doesn‟t matter whether I do it with the Chinese group among the three application domains.
a full hand or just with a single finger. I chose it The comparison of agreement scores between the two
because it‟s so cool, like a conductor raising his/her cultural groups among three application domains is shown
hand to increase the volume of the music”. in Figure 2.

In contrast, if the mental model of the task was different,

we considered the differences between hand shapes and/or
trajectories when merging gestures. For example, task V19
– Move object in the VR scenario, was performed by some
participants by pointing to the target and dragging it to the
new location using their right index finger. The participants
were inclined to borrow gestures that had already been used
in other designs (e.g., touch-based interfaces) instead of
designing new gestures.

“I chose this gesture because I used to do it on my phone

to move an object”.

A different interaction model for this task was found in

other participants: pointing to the target with the left index
finger while the right index finger points to the new
location. The participants imagined that they have magic
powers when interacting with the VR scenario; therefore,
they could leverage interactions beyond the real world:

“I chose this gesture with the underlying mental model (a) the Car scenario

that the selected target with the left index would fly
over the new location indicated by the right index
finger”.

Apparently, the two gestures had different mental models

behind them and couldn’t be arbitrarily merged.
After the grouping and merging process, the number of
gestures in both cultural groups were reduced. As a result,
we obtained 530 groups of identical gestures from the
American group and 439 groups from the Chinese group.

Agreement examination between two cultural groups

When measuring gesture agreement, we used a formula by
Vatavu et al. (2015), rather the formula by Wobbrock et al.
(2009), which is widely used by early research (Ruiz et al.,
2011; Obaid et al., 2012; Vatavu, 2012; Buchanan et al.,
2013; Kistler et al., 2013; Piumsomboon et al., 2013;
Rovelo et al., 2014; Valdes et al., 2014; Tung et al., 2015).
We made this choice because unlike the formula by
Wobbrock et al. which cannot well handle extreme (b) the VR scenario
situations, such as no single gesture obtaining more than
one vote, the measure by Vatavu et al. is more general and
is valid even for extreme cases.
The formula we used is defined as follows:
6
 gesture, selected by 7 Chinese participants but only by 4
Americans.
Next, we identified the top 20 target tasks that had the
same top gesture in both cultural groups and the agreement
score of these tasks were ranked in the top half among all
tasks in the same scenario and in the same cultural group.
Table 2 lists the 20 target tasks with high agreement and
the same top gestures in both cultural groups. The two left
columns of the table identify each task with its top gesture.
The two right columns indicate how many participants in
the American group (#A) and the Chinese group (#C) chose
a gesture for its corresponding target task.

Table 2
Top 20 target tasks shared the same top gestures between two cultural
groups.
Task Gesture #A #C
C3. Previous track Swipe left 13 11
C4. Next track Swipe right 13 11
C5. Volume up Swipe up 22 14
C6. Volume down Swipe down 22 14
(c) the TV scenario
Figure 2. Comparison of agreement scores between the American C15. Pick up the phone Hold the thumb near the ear and 20 16
group and the Chinese group among three application domains in point the pinky at the mouth
Experiment 1. C17. Hang up the phone Move the hand away from the 16 10
face while keeping the “Pick-
From Figure 2, we can see that, the patterns of the Up-The-Phone” gesture
mentioned above, and point
agreement scores for both cultural groups are largely
both the thumb and pinky
similarly in three application scenarios. For example, in the downward.
Car scenario tasks such as C25 – Zoom in the map and C26 C21. Temperature up Swipe up 17 15
– Zoom out the map have the highest scores in both cultural C22. Temperature down Swipe down 17 15
groups, while tasks such as V4 – Slow down in the VR C25. Zoom in the map Perform a pinch-out gesture with 22 23
scenario get the lowest scores in both groups. fingers
By statistically analyzing the agreement scores between C26. Zoom out the map Perform a pinch-in gesture with 22 23
the two cultural groups, we found that the average fingers
agreement scores of the 70 target tasks in American and V6. Turn left Swipe left 18 18
Chinese groups are 0.324 (SD = 0.227) and 0.290 (SD = V7. Turn right Swipe right 18 18
0.223), respectively. The Mann-Whitney U test indicated V10. Open the box Move two hands from the center 18 18
that no significant difference was found between the two to the left and the right,
cultural groups (Z = -1.005, p = .315). respectively
V11. Pick up an object Grab an imagined object and 19 19
Top gestures between two cultural groups take it out
As shown in Figure 2, the two groups have the highest V12. Close the box Move two hands from the left and 21 12
agreement scores on some tasks (e.g., C25 and C26) right to the middle
simultaneously, indicating high agreement on these tasks V20. Rotate object Grab an imagined object with a 18 11
hand and twist the hand
between two groups of participants. However, by taking a
V23. Copy object Make a fist with the left hand and 12 14
closer look at the top gestures for 70 target tasks between keep it still as the source
participants from two cultural groups, we found very location; then move the right
interesting results: for some tasks, such as C3 – Previous hand from the left-hand
track in the Car scenario, participants from two cultural location to a new location.
groups produced the same top gestures Swipe left, while for T7. Volume up Swipe up 19 21
some others, participants from two cultural groups T8. Volume down Swipe down 19 21
produced very different top gestures. For example, a static T17. Play a selected Use the index finger to click 19 19
“Stop” sign is the top gesture for C2 – Stop CD in the Car channel
scenario and was chosen by 10 American participants.
However, this gesture was only chosen by only 1 Chinese Discussion of Experiment 1
participant. In comparison, the top gesture by Chinese Agreement analysis results show that no significant
participants for the same task is a dynamic Fist close difference was found between the two cultural groups.
However, agreement analysis can only indicate the
7
consistency among the 70 tasks but cannot reflect the Agreement examination between two cultural groups
variations of top gestures for those tasks between two Similar to Experiment 1, we calculated the differences in
cultural groups. Therefore, we examined the top gestures agreement between the American group and the Chinese
for each task produced by participants from two groups. group among the three application domains by using the
Results show that some tasks had the same top gestures same agreement rate formula as shown in Equation 1. The
while some had totally different top gestures between two comparison of agreement scores between the two groups
cultural groups. Next, we identified the top 20 tasks that among three application domains is shown in Figure 3.
had high agreement and the same top gestures between two
groups. These 20 tasks can be regarded as gesture designs
where the cultural bias problem may be insignificant and
the remaining 50 tasks will be left for further analysis in the
following experiments.

3.2.2 Experiment 2
In Experiment 1, participants were asked to design gestures
for given target tasks without any constraint. Experimental
results imply the existence of the influence of cultural
factors on gesture design. To further explore this issue, we
designed a follow-up experiment. Different from the
previous experiment, participants in this experiment were
asked to choose a gesture for each of the 50 tasks that
indicate cultural difference in gesture choices.

Participants and apparatus

A new group of 48 participants, 24 from the USA and 24
from China, were recruited for this experiment. These
participants from the same university, respectively, as that
in Experiment 1 and with similar profiles and backgrounds
(a) the Car scenario
as those in Experiment 1. Among 24 American participants,
20 are males and 4 are females. Among 24 Chinese
participants, 10 are males and 14 are females. None of
these 48 participants had any experience with gesture
design and participated in Experiment 1. We used the same
experimental set-up as that in Experiment 1.

Procedure
Different from Experiment 1, participants in this
experiment were required to choose the best gesture for
each given target task verbally instead of designing a best
gesture for them.
During the experiment, participant were first briefly
introduced experimental objective, tasks and requirements,
and then were shown a set of gesture candidates collected
from all designs by participants from the same cultural
group in Experiment 1 for each task. After seeing all the
gesture candidates for a given task, participants were asked
to choose the best gesture for this task according to their
personal preferences. They were also required to explain (b) the VR scenario
verbally why a specific gesture was chosen for each target
task.

Results
In this experiment, the American cultural group were
provided 435 gestures to choose for the 50 target tasks and
the Chinese cultural group were provided 444 gestures.
After this experiment, the American and Chinese cultural
groups chose 256 and 294 gestures, respectively.
8
 participants in the American group (#A) and the Chinese
group (#C) chose a gesture for its corresponding target task.

Table 3
Top 14 target tasks shared the same top gestures between two cultural
groups.
Task Gesture #A #C
C9. Turn on radio Clockwise twist 10 12
C11. Next station Swipe right 14 14
C12. Previous station Swipe left 14 14
V15. Single selection Click 15 14
V16. Release object Open the fist 17 11
V18. Multiple selection Click objects one by one 14 11
V19. Move object Move object with index finger 14 12
V21. Enlarge object Pinch-out object with the whole 13 10
hand
V22. Shrink object Pinch-in object with the whole 13 10
hand
T5. Next channel Swipe right 14 13
T6. Previous channel Swipe left 15 14
(c) the TV scenario T11. Change to channel Draw “168” with index finger in 17 16
Figure 3. Comparison of agreement scores between the American 168 air
group and the Chinese group among three application domains in T12. Change to channel Draw “79” with index finger in 18 16
Experiment 2.
79 air
T13. Change to channel Draw “3” with index finger in 16 15
By statistically analyzing the agreement scores between 3 air
the two cultural groups, we found that the average
agreement scores of the 50 target tasks in American and
Chinese groups are 0.355 (SD = 0.133) and 0.304 (SD = Discussion of Experiment 2
0.141), respectively. The Mann-Whitney U test indicated Different from traditional elicitation studies, we provided
that significant difference was found between the two end-users a set of pre-designed gestures to choose in this
cultural groups (Z = -2.175, p = .030). section. Our aim is to further explore the influence of
cultural factors on gesture design.
Top gestures between two cultural groups Experimental results show that, 14 out of 50 given target
We examined the top gestures for 50 target tasks between tasks have the same top gestures between two cultural
participants from two cultural groups and found consistent groups. Combined with the top 20 tasks listed in Table 2,
results as in Experiment 1: for some tasks, such as V21 – we can thus obtain 34 tasks with high agreement and
Enlarge object in the VR scenario, participants chose the meanwhile have the same top gestures between two cultural
same top gestures Pinch-out object with the whole hand groups. By analyzing the characteristic of these 34 tasks,
from two cultural groups. While for some others, we divided them into 4 categories as shown in Table 4.
participants from two cultural groups selected different top
Table 4
gestures. For example, a static Thumb up is the top gesture 4 types of tasks that are less influenced by cultural bias problem.
for T4 – Confirm in the TV scenario, and was chosen by 14 Task type Tasks
American participants. However, this gesture was taken by C3. Previous track, C4. Next track,
only 8 Chinese participants. In comparison, the top gesture C5.Volume up, C6. Volume down,
preferred by Chinese participants for the same task is a C11. Next station, C12. Previous station,
Associated with C21.Temperature up, C22. Temperature
static “OK” gesture, selected by 13 Chinese participants but direction or order down, V6. Turn left, V7. Turn right,
only by 5 Americans. T5. Next channel, T6. Previous channel,
Based on the same criteria described in Experiment 1, we T7.Volume up, T8. Volume down
selected the top 14 target tasks that had the same top C25. Zoom in the map, C26. Zoom out the
gesture in both cultural groups and the agreement score of map, V15. Single selection, V16. Release
these tasks were ranked in the top half among all tasks in Related to object object, V18. Multiple selection, V19. Move
manipulation object, V20. Rotate object, V21. Enlarge
the same scenario and in the same cultural group. object, V22. Shrink object, V23. Copy
Table 3 lists 14 target tasks with high agreement and the object, T17. Play a selected channel
corresponding top gestures in both cultural groups. The two Dealing with objects C9. Turn on radio, C15. Pick up the phone,
left columns of the table identify each task with its top that can be mapped to C18. Hang up the phone, V9. Open the box,
gesture. The two right columns indicate how many concrete artifacts in the V11. Pick up an object, V12. Close the box
real world

9
Associated with aimed to further validate the impact of culture on freehand
symbols that are T11. Change to channel 168, T12. Change to gestures and to identify the types of tasks that could be
universally accepted channel 79, T13. Change to channel 3
(e.g., Arabic numerals)
influenced by cultural factors.

3.2.3 Experiment 3
For the other tasks, we saw significantly different top This experiment focused on those target tasks where
gestures between the two cultural groups. For example, a participants from two groups showed different preferences
Thumb up gesture is the top gesture for T4 – Confirm in the for gesture designs. Based on the results from the previous
TV scenario, and was chosen by 12 American participants. two Experiments, we identified 34 target tasks that had
However, this gesture was only chosen by 3 Chinese high agreement and the same top gesture between the two
participants. The top gesture preferred by Chinese groups. Among the remaining 36 tasks, 10 of them had the
participants for the same task is a static “OK” gesture, same top gestures by both American and Chinese groups,
selected by 6 Chinese participants but only by 4 Americans. although the agreement scores of these tasks were not
Another example is C18 – Reject a call: American ranked in the top half among all tasks in the same scenario
participants chose the gesture of Cutting throat, but and in the same cultural group. Considering the fact that the
Chinese participants preferred the gestures of Shaking the top gestures of these 44 tasks (34 + 10) are the same across
right hand. both cultural groups, we chose the rest 26 tasks, which had
In addition to the differences in gestures with semantic different top gestures between two groups, as the target
interpretations, we also found that Chinese and American tasks in this experiment. In addition to these 26 tasks, we
participants had different habits in expressing numbers. For also found that the two groups exhibited very different
example, American and Chinese participants used different styles in expressing numbers 3, 6, 7, 8, 9, and 10. Therefore,
hand shapes to indicate number 3. Moreover, American we also integrated gestures to express these 6 digits into the
gestures require two hands for numbers 6 to 10, while all target task set. Table 5 showed the 32 tasks involved in this
Chinese gestures involve only one hand. Such information experiment.
laid the foundation for our next experiment, which was
Table 5
32 target tasks in Experiment 3.
Target tasks Top gesture by American group Top gesture by Chinese group
C1. Play CD Insert a CD Finger snap
C2. Stop CD A “Stop” sign Make a fist
C7. Pause music Pointing up with index and middle fingers Perform a „Hush” gesture with index finger near the
mouth
C13. CD to radio Insert a CD, swipe left, then point forward Draw a clockwise semicircle in air
with index finger
C14. Radio to CD Point forward, swipe left, then insert a CD Draw a counterclockwise semicircle in air
C16. Switch to the previous call Move the hand away from the face while Move the hand away from the face while keeping the
keeping the “Pick-Up-The-Phone” gesture “Pick-Up-The-Phone” gesture mentioned above, then
mentioned above, then swipe left wave back and forth
C18. Reject a call Cutting throat Waving back and forth
C19. Turn on air conditioner Wiping sweat from forehead Open the fist
C20. Turn off air conditioner Cutting throat Make a fist
C23. Fan speed up Swipe up Fan oneself
C24. Fan speed down Swipe down Perform a pinch-in gesture with fingers
C27. Move the map Move the map with index and middle fingers Move the map with the whole hand
V2. Step back Two hands counterclockwise rotation Pointing back with index finger
alongside the body
V5. Stop A “stop” sign Perform a “T” sign with two hands
V8. Open the door Grab an imagined knob, twist, then pull back Push forward with right hand
V9. Close the door Grab an imagined knob and push forward Push forward with two hands
V13. Confirm Thumb up An “OK” sign
V14. Cancel Thumb down Performing an “X” sign with two hands
T1. Turn on the TV Press an imagined power button with thumb Finger snap
and index finger
T2. Turn off the TV Press an imagined power button with thumb Finger snap
and index finger
T3. Cancel Thumb down Performing an “X” sign with two hands
T4. Confirm Thumb up An “OK” sign
T9. Mute Perform a „Hush” gesture with index finger Perform a “Stop” gesture then reverse the hand
near the mouth
T10. Cancel mute Shake the index finger back and forth then Open the fist
make a “Hush” gesture
T18. Call main menu Draw an “M” letter in air Open the fist
T19. Return to main menu Two palms facing forward Swipe up then left
T20. Change to channel 3

10
 T21. Change to channel 6

T22. Change to channel 7

T23. Change to channel 8

T24. Change to channel 9

T25. Change to channel 10

Participants and apparatus

A new group of 48 participants were recruited for this
experiment. Among them, 24 were students (12 males and
12 females) from the same American university as in
Experiments 1 and 2, and the other 24 (11 males and 13
females) from the same Chinese University. Similar to the
previous two experiments, participants came from different
majors and professional backgrounds and their ages were
between 20 and 28. None of these 48 participants had any
experience with gesture design and none of them had
participated in the previous two experiments.
For the purpose of consistency, we used the same
experimental set-up as that in Experiment 2.

Procedure
Each test started with a brief introduction to the experiment
and consent process. Then, participants were asked to
choose a gesture for a list of 32 target tasks.
For each task, participants were given 2 gestures to
choose: one is the top gesture for this task by the American
group in Experiment 2, and the other by the Chinese group. FIGURE 4. Measurement of inter-rater reliability

A gesture was presented as an image (static gesture) or a

video clip (dynamic gesture). Text description was
accompanied with a gesture to explain a target task and its
interaction context. The order of the two gestures in each
task was counterbalanced.
After seeing the two gesture candidates for each task,
participants were asked to choose the best gesture based on
to what extent the gesture matched the task, how easy it is
to remember the gesture is, and how comfortable to
perform it. A voice recorder was used to record the reason
why participants chose a specific gesture for each task.
Results
In this section, we present the experimental results and
compare the two cultural groups from different perspectives.
As shown in Figure 4, the two groups showed very
inconsistent opinions on gesture choices for some tasks,
such as C18 – Reject a call and C20 – Turn off air
conditioner in the Car scenario.
Comparison of gesture preferences between two cultural
groups
Next, we used the Chi-Square test to examine whether there
is any difference in gesture preference between two groups.
The results were shown in Table 6.
From Table 6 we can see that significant difference in
gesture choice was found between two groups for 13 target
tasks, including C18, C20, V5, V13, V14, T3, T4, T20, T21,
T22, T23, T24 and T25 (p < .05).

Examination of consistency between two cultural groups
First, we used Randolph’s Kappa statistic (Randolph, 2005)
to measure the inter-rater reliability of gesture selections
between the two cultural groups. Figure 4 shows the Kappa
value obtained for each of the 32 target tasks.
Comparison of social acceptance of gestures by the same
and/or different cultural group
Next, we examined the social acceptance of gestures, i.e.,
whether the gestures were deemed to be appropriate in the
context in which they were carried out by participants
under the same and/or different cultural background.
For C18 – Reject a call, half American participants chose
Cutting throat, a gesture proposed by American participants
while the other half chose Waving back and forth suggested
by the Chinese group. In comparison, all 24 Chinese
participants chose Waving back and forth.

11
 For C20 – Turn off air conditioner, 17 American a fist suggested by Chinese group. In comparison, none of
participants chose the gesture of Cutting throat, which was 24 Chinese participants chose Cutting throat.
suggested by American group while the other 7 chose Make
Table 6
Chi square p value of the 32 target tasks
American participants Chinese participants
in Experiment 3 In Experiment 3
Task Top gesture by Top gesture by Top gesture by Top gesture by Sig.
American group Chinese group in American group Chinese group in
in Experiment 2 Experiment 2 in Experiment 2 Experiment 2
C1. Play CD 7 17 5 19 0.505
C2. Stop CD 18 6 18 6 1.000
C7. Pause music 5 19 8 16 0.330
C13. CD to radio 1 23 2 22 1.000
C14. Radio to CD 1 23 2 22 1.000
C16. Switch to the previous call 14 10 13 11 0.771
C18. Reject a call 12 12 0 24 0.000
C19. Turn on air conditioner 5 19 8 16 0.330
C20. Turn off air conditioner 17 7 0 24 0.000
C23. Fan speed up 8 16 5 19 0.330
C24. Fan speed down 12 12 18 6 0.074
C27. Move the map 8 16 12 12 0.242
V2. Step back 12 12 15 9 0.383
V5. Stop 24 0 13 11 0.001
V8. Open the door 7 17 6 18 0.745
V9. Close the door 8 16 9 15 0.763
V13. Confirm 21 3 4 20 0.000
V14. Cancel 13 11 3 21 0.006
T1. Turn on the TV 18 6 14 10 0.221
T2. Turn off the TV 16 8 15 9 0.763
T3. Cancel 13 11 3 21 0.006
T4. Confirm 21 3 4 20 0.000
T9. Mute 6 18 9 15 0.350
T10. Cancel mute 6 18 4 20 0.722
T18. Call main menu 7 17 11 13 0.233
T19. Return to main menu 8 16 3 21 0.170
T20. Change to channel 3 19 5 12 12 0.035
T21. Change to channel 6 24 0 0 24 0.000
T22. Change to channel 7 24 0 4 20 0.000
T23. Change to channel 8 24 0 2 22 0.000
T24. Change to channel 9 24 0 1 23 0.000
T25. Change to channel 10 24 0 3 21 0.000

For V5 – Stop, all 24 American participants chose A For T20 – Change to channel 3, 19 American participants
“Stop” sign designed by the American group while none of
them chose Perform an “T” sign with two hands designed chose designed by American culutral group while only
by Chinese group. In comparison, 13 Chinese participants 5 chose designed by Chinese cultural group. In
chose A “Stop” sign and the other 11 chose Perform a “T”
sign with two hands. comparison, half of 24 Chinese participants chose
For V13 and T4, Confirm, 21 American participants while the other half chose .
chose Thumb up designed by the American group and only For T21 – Change to channel 6, all 24 American
3 chose the “OK” sign designed by the Chinese group. In
comparison, only 4 Chinese participants chose Thumb up participants chose designed by the American group.
while 20 chose “OK”.
In comparison, all 24 Chinese participants chose
For V14 and T3, Cancel, 13 American participants chose
designed by Chinese cultural group.
Thumb down designed by the American group and the other
For T22 – Change to channel 7, all 24 American
11 chose Performing an “X” sign with two hands designed
by the Chinese group. In comparison, only 3 Chinese participants chose designed by the American
participants chose Thumb down while the other 21 chose group. In comparison, only 4 Chinese participants chose
Performing an “X” sign with two hands.
12

while the other 20 chose designed by Chinese
cultural group.
For T23 – Change to channel 8, all 24 American
participants chose designed by the American
group. In comparison, only 2 Chinese participants chose
while the other 22 chose designed by the
Chinese group.
For T24 – Change to channel 9, all 24 American
participants chose designed by the American
group. In comparison, only 1 Chinese participants chose
while the other 23 chose designed by Chinese
group.
For T25 – Change to channel 10, all 24 American
participants chose designed by the American group.
In comparison, only 3 Chinese participants chose
while the other 21 chose desigend by Chinese group.
Discussion of Experiment 3
In this experiment, we further verified the influence of
cultural factors on freehand gesture design. From Figure 4,
we observed different gesture preferences between the two
cultural groups for 32 target tasks. The Chi-Square test
results show that significant difference was found between
the gesture choices for 13 tasks.
Examining gesture choices for these 13 tasks, we can
classify them into two types. The first type of gestures
includes those strongly associated with specific cultural
semantics or symbols that may not be easily understood by
participants in other cultures. For example, the Cutting
throat gesture by American participants for C18 – Reject a
call and C20 – Turn off air conditioner was chosen by more
than a half American participants, but none Chinese
participants. Asked why this gesture was not selected, some
Chinese participants indicated that this gesture is seen as
killing a person. Another example is the “Perform a “T”
sign with two hands” gesture for V5 – Stop. None of 24
American participants chose this gesture because they all
saw the “Stop” sign as a universal gesture. This finding was
consistent with the report by Jane et al. (2017), who found
that Chinese participants preferred to use a T-shape gesture
for stopping the drone as compared to the American
participants. For gestures to indicate numbers larger than 5,
none of 24 American participants know the meaning of the
5 single-hand gestures as shown in Table 5.
Another type of gestures includes those that can be
understood and accepted by participants in other cultures in
spite of the significant difference in gesture preferences
between different cultural groups. For example, 21
American participants chose the “Thumb up” gesture for
V13 – and T4 – Confirm. When asked the reason why they
chose, participants indicated that the gesture of Thumb up
gives stronger approval than the one of OK. In comparison,
13
3 American participants chose the “OK” gesture for V13
and T4. They explained that it was just their personal
preferences. Similarly, 11 American participants can accept
the top gesture proposed by Chinese Cultural group
“Performing an “X” sign with two hands” for V14 and T3,
and 3 Chinese participants can accept the “Thumb down”
gesture proposed by American cultural group.
4 Discussion: implication for freehand gesture
design
Based on the results and analyses of the 3 experiments, we
derived several guidelines of freehand gesture design in
freehand-gesture-based interaction.
4.1 Be aware of the cultural bias on freehand gesture
design
Designers should keep in mind that in traditional gesture
elicitation studies, participants’ cultural background could
influence the design of their own gesture inputs to the
system. Our study discovered that some target tasks (Table
5) had totally different top gestures between the two
cultural groups and significant difference was found about
the gesture preferences for 13 tasks between American and
Chinese cultural groups. Some gestures are very popular in
one culture but totally unacceptable in another, as seen
from the Cutting throat gesture for stop command and
static single-hand gestures for digits (Table 5). Just as one
Chinese participant stated that:
“It‟s rarely to see a Cutting throat gesture people used in
daily life to communicate with other people in East
Asian culture because it is impolite and even
aggressive.”
In comparison, some gestures which are commonly used
in one culture can also be identified in another culture. For
example, some American participants are willing to use the
static “OK” sign favored by Chinese participants for V13 –
and T4 – Confirm, and some Chinese participants have no
objection to the use of Thumb down gesture that are very
common in America for V14 – and T3 – Cancel. Compared
to such gestures as Cutting throat, these gestures are less
sensitive to the influence of cultural factors. Just as one
Chinese participant stated that:
“It‟s OK to use Thumb down to express opposition to or
disagreement with something because it has been
standardized as an emoji and is commonly seen in social
media such as WeChat, QQ, and microblog in China.”
For these types of tasks, designers should identify such
nuance of meaning in different cultures when designing and
using gestures.
4.2 Sensitivity of gesture to cultural bias
The influence of cultural factors varies from task to task.
While we found similar or even identical gestures for some
tasks from participants in both cultures, obvious differences
in gesture choice were also observed between two groups.
4.2.1 Tasks less sensitive to cultural bias
For some tasks, there was high agreement both within and
across the two cultures. For example, participants from
both cultures proposed the willingness to use multi-modal
interaction which integrates voice and gesture commands.
They noted that voice is more natural than gesture in some
cases such as the channel switch tasks (T11, T12, and T13)
in the TV scenario. Participants stated that:
“I‟d like to say a channel switch command while pointing
to the TV using the index finger when I perform this
type of tasks because it‟s convenient and effective.”
Our study identified four types of tasks where finding
commonly accepted top gestures by both cultural groups
may be less challenging, including
 tasks strongly associated with direction or order, such
as C3 – Previous track and C4 – Next track in the Car
scenario；
 tasks related to object manipulation, such as V19 –
Move object in the VR scenario；
 tasks dealing with objects that can be mapped to
concrete objects in the real world, such as C15 – Pick
up the phone in the Car scenario; and
 tasks associated with symbols that are universally
accepted, such as some Arabic numerals (dynamic
gestures).
For these types of tasks, designers may consider general
gesture designs that are acceptable by users in different
cultures.
4.2.2 Tasks more sensitive to cultural bias
Our study indicates two types of cultural bias that may
contribute to the significant difference in gesture choice.
The first factor is the cultural norm in gesture
expression, such as how many hands and what fingers
are involved in a gesture. In Chinese culture, only one
hand is used for any number not larger than 10, while in
American culture, two hands are used to show numerals
larger than five. Even for numbers that can be expressed
with one hand, different fingers are used in different
cultures. The second factor is related to language. For
some tasks, in particular those tasks concerning abstract
concept, users from different cultures may have different
mental models and therefore choose different gestures.
For example, we observed that the language patterns by
two groups for tasks related to changing a channel are
different. While most American participants simply spoke
the words “change to [channel …]” when they performed a
gesture command, Chinese participants tended to choose
different vocabulary, including “change to …”, “switch
14
to …”, and “jump to …”. The diversity in vocabulary
choice leads to low agreement in gestures for related tasks.
Further study is needed to confirm the impact of such
factors on gesture choice and identify other possible
culture-related factors.
5 Conclusion and future work
In this work, we conducted a series of experiments to
investigate the influence of cultural factors on freehand
gesture design across three different application domains –
Car, VR and TV, with participants from two different
cultures – the USA and China. The main contribution of
this work include:
 we provided empirical evidence showing the existence
of cultural bias on freehand gesture interaction, of
which designers should be aware in gesture design;
 we discovered cultural similarities and differences in
the use of freehand gestures in different application
domains, among which 34 gestures are less influenced
by cultural factors while 13 gestures are significantly
influenced by participants’ cultural background;
 we identified the types of tasks that exhibit significant
difference between cultures, as well as the types of
tasks that are less sensitive to cultural factors; and
 we derived some design guidelines for freehand
gesture design, which we hope can also be informative
to other related gesture-based design.
There are some limitations of this study. One limitation
is that our study only involved two cultural groups. To
further generalize our findings, more research is need to
investigate whether the same conclusions can still be held
when users from other cultures are involved. In addition,
the participants recruited in this study were all college
students. More work is needed in the future involving
participants of different ages and occupations to generalize
our conclusions. Also, although our research shows the
effect of culture on end-users’ preferences of gesture
commands, our experiments cannot reveal what specific
cultural factors may cause the difference between users
from two cultures. Further study is needed to investigate
what underlying cultural causes may play a role.
In addition to our findings on the disagreement on
gesture commands between groups from two cultures, we
also observed some interesting phenomena that demand
further research. For example, we found that in both groups,
the user agreement score on the task “Moving the map”
when using a map inside a car, a more commonly seen and
less abstract task, is lower than the scores on the tasks
“Zooming in/out the map”, a relatively less common but
more abstract task. More study is needed to know what
cognitive or behavioral factors may lead to such a counter-
intuitive result.
Acknowledgements
The authors would like to thank the anonymous reviewers
for their insightful comments. This work was supported by
the National Natural Science Foundation of China
(61772564, 61772468) and the National Key Research and
Development Program of China (2018YFB1004903).
Disclosure statement
No potential conflict of interest was reported by the authors.
References
Alkemade, R.,Verbeek, F.J., Lukosch, S.G. (2017). On the
efficiency of a VR hand gesture-based interface for 3D object
manipulations in conceptual design. International Journal of
Human-Computer Interaction. 33(11). pp. 882-901.
Angelini, L., Carrino, F., Carrino, S., Caon, M., Khaled, O.A.,
Baumgaartner, J., Sonderegger, A., Lalanne, D., Mugellini,
E. (2014). Gesturing on the steering wheel: a user-elicited
taxonomy. In: AutomotiveUI’14. pp: 1-8.
Brown, A. (2008). Gesture viewpoint in Japanese and English:
cross-linguistic interactions between two languages in one
speaker. Gesture, 8. pp. 256-276.
Buchanan, S., Floyd, B., Holderness, W., LaViola, J.J. (2013).
Towards user-defined multi-touch gestures for 3D objects. In:
ITS’13. pp. 231-240.
Cai, Z.Y., Han, J.G., Liu, L., Shao, L. (2017). RGB-D datasets
using Microsoft Kinect or similar: a survey. Multimedia
Tools and Applications. 76. pp. 4313-4355.
Chan, E., Seyed, T., Stuerzlinger, W., Yang, X.D., Maurer, F.
(2016). User elicitation on single-hand microgestures. In:
CHI’16. pp. 3403-3414.
Chen, Z., Ma, X.C., Peng, Z.Y., Zhou, Y., Yao, M.G., Ma, Z.,
Wang, C., Gao, Z.F., Shen, M.W. (2018). User-defined
gestures for gestural interaction: extending from hands to
other body parts. International Journal of Human-Computer
Interaction. 34(3). pp. 238-250.
Cheng, H., Yang, L., Liu, Z.C. (2016). Survey on 3D hand gesture
recognition. IEEE Transactions on circuits and systems for
video technology. 26(9). 1659-1673.
Connell, S., Kuo, P.Y., Liu, L., Piper, A.M. (2013). A Wizard-of
Oz elicitation study examining child-defined gestures with a
whole-body interface. In: IDC’13, pp. 277-280.
Döring, T., Kern, D., Marshall, P., Pfeiffer, M., Schöning, J.,
Gruhn, V., Schmidt, A. (2011). Gestural interaction on the
steering wheel – reducing the visual demand. In: CHI’11.
pp. 483-492.
Dong, H.W., Danesh, A., Figueroa, N., Saddik, A.E. (2015). An
elicitation study on gesture preferences and memorability
toward a practical hand-gesture vocabulary for smart
televisions. IEEE Access. pp. 543–555.
Feng, Z.Q., Yang, B., Li, Y., Zheng, Y.W., Zhao, Z.Y., Yin, J.Q.,
Meng, Q.F. (2013). Real-time oriented behavior-driven 3D
freehand tracking for direct interaction. Pattern Recognition.
46(2). pp. 590-608.
Furnas, G.W., Landauer, T.K., Gomez, L.M., Dumais, S.T.
(1987). The vocabulary problem in human-system
communication. Communications of the ACM. 30(11). pp.
964-971.
Gheran, B.F., Vanderdonckt, J., Vatavu, R.D. (2018). Gestures for
smart rings: empirical results, insights, and design
implications. In: DIS’18. pp. 623-635.
15
Grijincu, D., Nacenta, M.A., Kristensson. P.O. (2014). User-
defined interface gestures: dataset and analysis. In: ITS‟14.
pp. 25-34.
Han, J.G., Shao, L., Xu, D., Shotton, J. (2013). Enhanced
computer vision with Microsoft Kinect sensor: a review.
IEEE Transactions on Cybernetics. 43. 5. pp. 1318-1334.
Hilliges, O., Izadi, S., Wilson, A.D., Hodges, S., Mendoza, A.G.,
Butz, A. (2009). Interactions in the air: adding further depth
to interactive tabletops. In: UIST’ 09. pp. 139–148.
Hoff, L., Hornecker, E., Bertel, S. (2016). Modifying gesture
elicitation: do kinaesthetic priming and increased production
reduce legacy bias? In: TEI’16. pp. 86-91.
Jane, L.E., Ilene, L.E., Landay, J.A., Cauchard, J.R. Drone & Wo:
Cultural Influences on Human-Drone Interaction Techniques.
In: CHI‟17. pp. 6794-6799.
Kistler, F., André, E. (2013). User-defined body gestures for an
interactive storytelling scenario. Interact ’13. pp. 264-281.
Kray, C., Nesbitt, D., Rohs, M. (2010). User-defined gestures for
connecting mobile phones, public displays, and tabletops. In:
MobileHCI‟10. pp. 239-248.
Kristensson, P.O., Nicholson, T.F.W., Quigley, A. (2012).
Continuous recognition of one-handed and two-handed
gestures using 3D full-body motion tracking sensors. In:
IUI‟12. 89-92.
Kulshreshth, A., LaViola Jr., J.J. (2014). Exploring the usefulness
of finger-based 3D gesture menu selection. In: CHI’14, pp.
1093-1102.
Kurdyukova, E., Redlin, M., André, E. (2012). Studying user-
defined iPad gestures for interaction in multi-display
environment. In: IUI‟12. pp. 93-96.
Kühnel, C., Westermann, T., Hemmert, F., Kratz, S. (2011). I’m
home: defining and evaluating a gesture set for smart-home
control. International Journal of Human-Computer Studies.
69. pp.693-704.
Lee, G.A., Choi, J.S., Wong, J., Park, C.J., Park, H.S., Billinghurst,
M. (2015). User defined gestures for augmented virtual
mirrors: a guessability study. In: CHI’15. pp. 959-964.
Lou, X.L., Peng, R., Hansen, P., Li, X.D.A. (2018). Effects of
user’s hand orientation and spatial movements on free hand
interactions with large displays. International Journal of
Human-Computer Interaction. 34(6). pp. 519-532.
Löcken, A., Hesselmann, T., Pielot, M., Henze, N., Boll, S.
(2011). User-centered process for the definition of free-hand
gestures applied to controlling music playback. Multimedia
Systems. 18(1). pp. 15-31.
Mauney, D., Howarth, J., Writanen, A., Capra, M. (2010).
Cultural similarities and differences in user-defined gestures
for touchscreen user interfaces. In: CHI’10. pp. 4015-4020.
Morgado, L. (2013). Cultural awareness and personal
customization of gestural commands using a shamanic
interface. In: DSAI’13. pp. 449-459.
Morris, M.R., Wobbrock, J.O., Wilson, A.D. (2010).
Understanding users’ preferences for surface gestures. In:
GI‟10. pp. 261-268.
Morris, M.R. (2012). Web on the wall: insights from a multimodal
interaction elicitation study. In: ITS’12. pp. 95-104.
Morris, M.R., Danielescu, A., Drucker, S., Fisher, D., Lee, B.,
Schraefel, M.C., Wobbrock, J.O. (2014). Reducing legacy
bias in gesture elicitation studies. Interactions. 21(3). pp. 40-
45.
Nacenta, M.A., Kamber, Y., Qiang, Y.Z., Kristensson, P.O.
(2013). Memorability of pre-designed & user-defined gesture
sets. In: CHI‟13. pp. 1099-1108.
Nebeling, M., Huber, A., Ott, D., Norrie, M.C. (2014). Web on
the wall reloaded: implementation, replication and
refinement of user-defined interaction sets. In: ITS’14. pp.
15-24.
Obaid, M, Häring, M., Kistler, F., Bühling, R., André, E. (2012).
User-defined body gestures for navigational control of a
humanoid robot. In: ICSR’12. pp. 367-377.
Peshkova, E., Hitz, M., Ahlström, D., Alexandrowicz, R.W.,
Kopper, A. (2017). Exploring intuitiveness of metaphor-
based gestures for UAV navigation. In: 26th IEEE
International Symposium on Robot and Human Interactive
Communication (RO-MAN). pp. 175-182.
Pfeil, P., K. Koh, S.L., LaViola Jr., J.J. (2013). Exploring 3D
gesture metaphors for interaction with unmanned aerial
vehicles. In: IUI’ 13. pp. 257- 266.
Piumsomboon, T., Billinghurst, M., Clark, A., Cockburn, A.
(2013). User-defined gestures for augmented reality.
CHI‟13. pp. 955-960.
Randolph, J.J. (2005). Free-marginal multirater kappa: An
alternative to Fleiss´ fixed-marginal multirater kappa. Online
Submission. 4(3):20.
Rautaray, S.S., Agrawal, A. (2015). Vision based hand gesture
recognition for human computer interaction: a survey.
Artificial Intelligence Review. 43. pp.1-54.
Rehm, M., Bee, N., André, E. (2008). Wave like an Egyptian:
accelerometer based gesture recognition for culture specific
interactions. In: BCS-HCI’08. pp. 13-22.
Rovelo, G., Vanacken, D., Luyten, K., Abad, F., Camahort, E.
(2014). Multi-viewer gesture-based interaction for omni-
directional video. In: CHI‟14. pp. 4077-4086.
Ruiz, J., Li, Y., Lank, E. (2011). User-defined motion gestures
for mobile interaction. In: CHI‟11.pp.197-206.
Serrano, M., Ens, B., Irani, P. (2014). Exploring the use of hand-
to-face input for interacting with head-worn displays. In:
CHI’14. pp. 3181-3190.
Seyed, T. Burns, C., Sousa, M.C. Maurer, F., Tang, A. (2012).
Eliciting usable gestures for multi-display environments. In:
ITS’12. pp. 41-50.
Shimon, S.S.A., Lutton, C., Xu, Z.C., Smith, S.M., Boucher, C.,
Ruiz, J. (2016). Exploring non-touchscreen gestures for
smartwatches. In: CHI’16. pp. 3822-3833.
Song, P., Goh, W.B., Hutama, W., Fu, C.W., Liu,X.P. (2012). A
handle bar metaphor for virtual object manipulation with
mid-air interaction. In: CHI’12. pp. 1297-1236.
Takahashi, M., Fujii, M., Naemura, M., Satoh, S. (2013). Human
gesture recognition system for TV viewing using time-of-
flight camera. Multimedia Tools and Applications. 62(3). pp.
761-783.
Tian, F., Lyu, F., Zhang, X.L., Ren, X.S., Wang, H.A. (2017). An
empirical study on the interaction capability of arm
stretching. International Journal of Human-Computer
Interaction. 33(7). pp. 565-575.
Turk, M. (2014). Multimodal interaction: a review. Pattern
Recognition Letters. 36. pp. 189-195.
Tung, Y.C., Hsu, C.Y., Wang, H.Y., Chyou, S., Lin, J.W., Wu,
P.J., Valstar, A., Chen, M.Y. (2015). User-defined game
16
input for smart glasses in public space. In: CHI’15. pp. 3327-
3336.
Valdes, C., Eastman, D., Grote, C., Thatte, S., Shaer, O., Mazalek,
A., Ullmer, B., Konkel, M.K. (2014). Exploring the design
space of gestural interaction with active tokens through user-
defined gestures. In: CHI‟14. pp. 4107-4116.
Vatavu, R.D. (2012). User-defined gestures for free-hand TV
control. In: EuroITV‟12. pp. 45-48.
Vatavu, R.D., Wobbrock, J.O. (2015). Formalizing agreement
analysis for elicitation studies: new measures, significance
test, and toolkit. In: CHI‟15.pp. 1325-1334.
Vatavu, R.D., Wobbrock, J.O. (2016). Between-subjects
elicitation studies: formalization and tool support. In:
CHI‟16. pp. 3390-3402.
Wang, X., Bernardos, A.M., Besada, J.A., Metola, E., Casar, J.R.
(2015). A gesture-based method for natural interaction in
smart spaces. Journal of ambient and smart environments. 7.
pp. 535-562.
Wobbrock, J.O., Aung, H.H., Rothrock, B., Myers, B.A. (2005).
Maximizing the guessability of symbolic input. In: CHI‟05.
pp. 1869-1872.
Wobbrock, J.O., Morris, M.R., Wilson, A.D. (2009). User-defined
gestures for surface computing. In: CHI‟ 09.pp.1083-1092.
Wu, H.Y., Wang, J.M., Zhang, X.L. (2016). User-centered gesture
development in TV viewing environment. Multimedia Tools
and Applications. 75. pp.733-760.
Wu, H.Y., Zhang, S.K., Liu, J.Y., Qiu, J.L. Zhang, X.L. (2018).
The gesture disagreement problem in freehand gesture
interaction. International Journal of Human-Computer
Interaction. https://doi.org/10.1080/10447318.2018.1510607.
Yang, H.D., Park, A.Y., Hee, S.W. (2007). Gesture spotting and
recognition for human-robot interaction. IEEE Transactions
on Robotics. 23(2).pp. 256-270.
Zaiţi, I.A., Pentiuc, Ş.G., Vatavu, R.D. (2015). On free-hand TV
control: experimental results on user-elicited gestures with
leap motion. Personal and Ubiquitous Computing. 19(5-6).
pp. 821-838.
Huiyue Wu received the Ph.D. degree in
computer science from Institute of Software, the
Chinese Academy of Sciences, China, in 2010.
He is currently an Associate Professor at Sun
Yat-sen University, Guangzhou, China. His
research interests include vision-based interfaces,
gestural interaction, and user-centered design.
Jinxuan Gai is a graduate student at
Pennsylvania State University, University Park,
PA. His research interests include human-
computer interaction and data visualization. He
obtained Bachelor of Engineering from Beijing
University of Technology, Beijing, China, 2014.
Yu Wang received the Ph.D degree from Sun
Yat-sen University, Guangzhou, China, in 2012.
She is an attending doctor at Sun Yat-sen
University Cancer Center, Guangzhou, China.
Her research interests include medical oncology, medical statistics, and
usability engineering.

Jiali Qiu is a graduate student at Sun Yat-sen

University, Guangzhou, China. Her research
interests include human-computer interaction,
interaction design, and usability engineering.
She obtained Bachelor of Design Science from
Sun Yat-sen University, Guangzhou, China,
2016.

Jiayi Liu is a graduate student at Sun Yat-sen

University, Guangzhou, China. Her research
interests include human-computer interaction,
interaction design, and usability engineering.
She obtained Bachelor of Design Science from
Sun Yat-sen University, Guangzhou, China,
2016.

Jianmin Wang received the Ph.D. degree from

Sun Yat-sen University, in 2003. He is currently
a full professor at the College of Arts & Media,
Tongji University. His research interests are in
the area of human-computer interaction and
interaction design.

Xiaolong (Luke) Zhang received the Ph.D.

degree from University of Michigan, Ann Arbor,
Michigan, in 2003. He is currently an associate
professor at the College of Information Sciences
and Technology, Pennsylvania State University.
His research interests are in the area of human-
computer interaction and focus on multiscale
technology, information visualization, virtual
environments, etc.

17
Credit Author Statement

Huiyue Wu: Conceptualization, Methodology, Validation, Formal analysis, Investigation, Writing –

original draft. Jinxuan Gai: Investigation. Yu Wang: Formal analysis. Jiayi Liu: Investigation. Jiali
Qiu: Investigation. Jianmin Wang: Validation. Xiaolong (Luke) Zhang: Conceptualization,
Methodology, Writing – review and editing

18