See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/349896617

Smooth Operator: Tuning Robot Perception Through Artificial Movement Sound

Conference Paper · March 2021

DOI: 10.1145/3434073.3444658

CITATIONS READS
2 24

3 authors, including:

Frederic Anthony Robinson

UNSW Sydney
8 PUBLICATIONS   10 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

GePS - Gesture-based Performance System View project

All content following this page was uploaded by Frederic Anthony Robinson on 24 May 2021.

The user has requested enhancement of the downloaded file.

Session 1: Motion
Smooth Operator: Tuning Robot Perception Through Artificial
Movement Sound
Frederic Anthony Robinson Mari Velonaki
Creative Robotics Lab Creative Robotics Lab
University of New South Wales University of New South Wales
Sydney, Australia Sydney, Australia
frederic.robinson@unsw.edu.au mari.velonaki@unsw.edu.au
ABSTRACT
Can we influence how a robot is perceived by designing the sound of
its movement? Drawing from practices in film sound, we overlaid
a video depicting a robot’s movement routine with three types
of artificial movement sound. In a between-subject study design,
participants saw either one of the three designs or a quiet control
condition and rated the robot’s movement quality, safety, capability,
and attractiveness. We found that, compared to our control, the
sound designs both increased and decreased perceived movement
quality. Coupling the same robotic movement with different sounds
lead to the motions being rated as more or less precise, elegant, jerky,
or uncontrolled, among others. We further found that the sound
conditions decreased perceived safety, and did not affect perceived
capability and attractiveness. More unrealistic sound conditions led
to larger differences in ratings, while the subtle addition of harmonic
material was not rated differently to the control condition in any
of the measures. Based on these findings, we discuss the challenges
and opportunities regarding the use of artificial movement sound
as an implicit channel of communication that may eventually be
able to selectively target specific characteristics, helping designers
in creating more refined and nuanced human-robot interactions.
CCS CONCEPTS
• Human-centered computing → Interface design prototyp-
ing; Auditory feedback.
KEYWORDS
human-robot interaction, sonic interaction design, movement soni-
fication
ACM Reference Format:
Frederic Anthony Robinson, Mari Velonaki, and Oliver Bown. 2021. Smooth
Operator: Tuning Robot Perception Through Artificial Movement Sound. In
Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot
Interaction (HRI ’21), March 8–11, 2021, Boulder, CO, USA. ACM, New York,
NY, USA, 10 pages. https://doi.org/10.1145/3434073.3444658
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
HRI ’21, March 8–11, 2021, Boulder, CO, USA
© 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-8289-2/21/03. . . $15.00
https://doi.org/10.1145/3434073.3444658
53
HRI ’21, March 8–11, 2021, Boulder, CO, USA
Oliver Bown
Interactive Media Lab
University of New South Wales
Sydney, Australia
o.bown@unsw.edu.au
1 INTRODUCTION
Social robots are moving from laboratories and exhibitions into
social environments such as homes, hospitals, nursing homes, and
public spaces. This places growing importance on rich and effective
human-robot interaction (HRI), as machines need to show engag-
ing behaviour to enable successful long-term interaction [6]. While
most sound-related work in the field of HRI focuses on speech and
nonverbal utterances, robot sound as an implicit channel of com-
munication has been receiving growing attention over recent years.
A useful definition of implicit communication for this context is
the one proposed by Breazeal et al., describing it as information,
which is "inherent in behaviour but which is not deliberately com-
municated." [7, p. 708]. In this context, one focal point [40, 48] is
consequential sound, defined as “a consequence of the moving parts
of products” [53, p. 69]. In robots this can be, for example, the sound
of a motor. A second aspect is movement sonification [17], which
uses sound as an accompaniment to movement with the aim of
conveying information. A growing body of research indicates how
our perception and expectations of robotic agents are influenced by
a robot’s inherent sound profile [17, 39, 40, 48, 51]. However, the
sounds used as stimuli are predominantly recordings of existing
servomotors, and a recurring suggestion is the design of artificial
sound to accompany, mask, or enhance robot motion. Describing
the communicative potential of robotic gesture, Hoffman and Ju
mention how "well-designed robot motion can communicate, en-
gage, and offer dynamic possibilities beyond the machines’ surface
appearance or pragmatic motion paths" [20, p. 89]. Given the im-
plicit impact of sound and the relevance of robot motion to HRI, it
appears promising to aim and treat movement sound as a design
space that can be used to enrich and refine HRI in specific and tar-
geted ways. We therefore set out to explore how accompanying a
robot’s motion with artificial sound can affect people’s impressions
and expectations of the robot as a whole.
In this paper, we describe the creation of three artificial sound
conditions, designed with varying degrees of realism and using the
broad sonic range available to sound designers. We then present
an online study investigating how these three sound designs and
a control condition influence participants’ ratings across various
measures relating to human perception of robots. We conclude with
our findings and discuss the challenges and opportunities of using
artificial movement sound as a communication channel in HRI.
2 RELATED WORK
HRI research on the effects of sound predominantly focuses on
various forms of utterances: sounds, which are deliberately com-
municated to convey affect or intention. For a complete review of
Session 1: Motion
semantic-free utterances in HRI, see [58]. But even if sound is not
used to intentionally communicate in an interaction, it is always
present and always impacts the listener. This section will give an
overview of the HRI research investigating consequential sound
and movement sonification.
2.1 Consequential sound
Several HRI studies have reported that consequential sound affected
participants’ perceptions. In an experiment by Trovato et al., partic-
ipants reported feeling "unsafe" when hugging a robot, due to the
noise emitted from its hand [50]. Inoue et al. found indicators that
the motor sound of social robot Paro negatively impacted interac-
tions [23]. Moore et al. reported a general aversion to engine noise
[40]. In a study examining affective gestures and their associated
servo sounds, Frid et al. showed possible problems of uncontrolled
sound, stating that "sounds inherent to robot movement can com-
municate other affective states than those originally intended to be
expressed" [17, p. 9]. This notion is supported by recent research
that shows how motor sound affects the perception of characteris-
tics like trustworthiness [39], strength and precision [40] or com-
petence [48]. This power of sound to implicitly convey information
and shape how humans perceive their environment through phys-
ical associations is well established in other disciplines, as is the
practice of actively using this communication channel. In product
sound design, communicated characteristics are often not noticed
by the user, but sub-consciously perceived. Car doors communicate
safety [61], the click of a luxury lighter communicates quality [31],
and bottles communicate freshness when opened [47]. In a process
called “active sound design” (ASD), firms like BMW and Renault
have added recorded and synthesised sound material to their cars’
engine sound to enhance the user experience. In electric vehicles
such as the Jaguar I-Pace, ASD has moved from augmenting exist-
ing engine sound to fully replacing it. In the case of autonomous
vehicles, ASD can additionally provide a communication channel
for information usually conveyed by the driver [38]. There are in-
stances, where auditory cues directly influence the perception of
other senses. In an experiment by Jousmäki and Hari, participants
rubbing their hands together reported having “dryer hands” when
the auditory feedback to their movements had a brighter spectrum
[28]. Watanabe and Shimojo showed how the movement of two
objects that could be visually interpreted as both “bouncing off“ or
“passing through” each other, was reliably categorised depending
on which sound they were paired with [54].
Working towards using consequential sound as a design space in
HRI, Frid et al. suggest masking or enhancing consequential sound
[17]. They suggest using what Tünnermann et al. define as blended
sonification: The process of changing the perception of a sound
event by adding sound elements that contain additional perceptual
information in a way that preserves the coherence of the original
sound [52]. This proposed masking was implemented in a study by
Trovato et al. who blended simulated consequential sound with a
musical soundscape [51]. They investigated whether this enhance-
ment of robotic sound would influence how close participants will
comfortably approach a robot. They found no significant impact
on participant distance, but reported that negative impressions of
54
HRI ’21, March 8–11, 2021, Boulder, CO, USA
the robot resulting from the untreated sound condition were ab-
sent when the sound was masked. An early example of designed
consequential sound can be found in Johannsen’s work [24–26].
He combined his musical utterances with recordings from various
robots to make use of the consequential sounds’ ability to inform
users about the robot’s location. The recordings were subsequently
filtered and pitched to better fit with the musical material. Another
application of designed consequential sound was presented by Cha
and colleagues, who processed audio recordings of the TurtleBot
servo motors and integrated them into a robotic agent to perform
a collaborative task with a human [9]. They found that adding a
masking noise to the robot’s sound profile made it easier to localize
and less disturbing.
2.2 Movement Sonification
Kramer et al. broadly define sonification as the use of non-speech
audio to convey information [30]. While the term movement sonifi-
cation is commonly used in the context of making human movement
audible [13], we will define it here as designing sound as an accom-
paniment to robot movement with the aim of conveying information.
Few studies have examined the sonification of expressive robotic
movement, although it is an active area of research in the SONAO
project [17]. Dahl et al. investigated connections between move-
ment quality and sound characteristics by analysing how trained
musicians sonify robotic movement [12]. This work later resulted
in a sound synthesis framework [4]. Latupeirissa et al. analysed
the sonic profile of several robots in popular science fiction films
and suggested their characteristics can be used as inspiration for
the auditory communication of internal states and the sonifica-
tion of gestures [34]. They later conducted a pilot study involving
mainly expert musicians to investigate the effect of various move-
ment sounds on the affective cues of robot Pepper. They found that
some sound conditions shifted participants’ interpretations of the
affective content of the gestures [33]. None of these mentioned im-
plementations were validated using larger-scale perceptual studies.
In the domain of semantic-free utterances, researchers have
aimed to synchronize robot movement with expressions, which
can be viewed as a form of movement sonification. Bramas et al.
proposed a system that synchronizes robotic motion with various
utterances ranging from music extracts and sound effects to voice
samples to increase the emotional impact of affective gestures [5].
Their study did not include a user evaluation. This notion of increas-
ing impact through synchrony can be connected to recent work
by Thiessen et al. who investigated the effect of infrasound on a
NAO robot’s affective communication and found that it led to some
emotions being perceived as more intense, despite participants not
being aware of the infrasound’s presence [49]. A technical imple-
mentation presented by Schwenk and Arras generated semantic-
free utterances in real-time, based on the movement data of their
robot Daryl [46]. They used what they termed motion modulation to
control pitch and tone of the robot’s utterances. This architecture al-
lowed them to easily link expressive gestures like body, head, or ear
pose with expressive utterances. Another example of a robot blend-
ing semantic-free utterances with motion is the toy robot Vector
by San Francisco-based consumer robot manufacturer Anki. Their
designers filed a patent that describes enhancing the consequential
Session 1: Motion
sound of their robot [56]. Their "condition-based audio techniques"
described syncing sound content to Vector’s movement and gesture
repertoire. The sound material was additionally dependent on the
robot’s moods, meaning that the movement of an excited robot
would sound differently from the movement of a disappointed one.
3 METHODOLOGY
Taking inspiration from an experimental design by Tennent et al.
[48], we designed a between-subject study to investigate the extent
to which artificial movement sound (the independent variable) can
influence how people perceive a robot’s movement, safety, capabil-
ity, and attractiveness (the dependent variables). We created three
sound designs and a quiet control condition for a video depicting
a robot in motion. After being randomly assigned to one of the
four conditions, participants watched the video and rated the robot
across the various measures in an online survey.
3.1 Task and Video Recording
Figure 1: Five excerpts of the movement routine shown in the video.
The video showed commercial robot Fetch perform a simple
movement routine (see Figure 1). Throughout the 20-second video
it first approaches the camera and adjusts its arm position from
slightly extended to tucked-in. It subsequently turns around and
moves away from the camera, before eventually turning to the left
and leaving the frame.1 We chose to use footage of Fetch as its
embodiment can be considered a hybrid between industrial robot
and humanoid. As some of our questions related to household tasks,
we wanted to show a robot that can be imagined in a home. The
actions performed by the robot were deliberately chosen to be
ambiguous and to allow different interpretations. Approaching the
camera may or may not indicate intentions, and tucking in the arm
may or may not be a communicative gesture such as a wave. By
keeping the behaviour of the robot relatively vague, we aimed to
investigate how sound influences these interpretations.
3.2 Sound Design
Our practice-based sound design approach is informed by a Re-
search through Design (RtD) methodology, an established frame-
work within the human-computer interaction community, which
enables designers to create prototypes and systems which "make
research contributions based on their strength in addressing under-
constrained problems" [59, p. 490]. Gaver notes its strength to create
"new, conceptually rich artefacts" [18, p. 937]. RtD is mentioned
as a potentially valuable approach to HRI research in a growing
number of publications [21, 22, 35, 41], with Luria et al. stating how
1A video with all sound conditions is available in the ACM digital library.
55
HRI ’21, March 8–11, 2021, Boulder, CO, USA
it can "enable researchers to conduct design work that questions
underlying assumptions and expands the boundaries of HRI" [36,
p. 4]. HRI researchers are also beginning to draw from the insights
of sound designers involved in the design of commercial robots
[43].
3.2.1 Sound Design Approach. We designed three sound condi-
tions (see Figure 2) and a control to be combined with the video
footage. These are subsequently referred to as mechanical, har-
monic, musical, and silent. Rather than using recordings of ex-
isting robots, we designed the conditions to cover a broad sonic
spectrum, drawing from the first author’s background in film sound.
A multidisciplinary overview of the various concepts and methods
applied in sound design for linear media can be found in [15]. As
part of their ongoing investigation into the sound of robots in film,
Latuperissa and Bresin suggest three categories: inner workings,
communication of movement, and expression of emotion [32]. Our
sound designs exclusively focus on the former two. To commu-
nicate inner workings, we designed sounds to represent internal
mechanics associated to specific actions, such as imaginary gears
moving into place before the beginning of a new motion sequence.
To communicate movement, we focused on start and end points of
motion sequences, either accentuating them with short bursts of
sound or distracting from them through gradual changes in volume
and tone color, or by not adding sound at all.
Figure 2: Frequency spectra of the three artificial sound conditions through-
out the video clip. Brighter colors indicate higher volume.
3.2.2 Sound Condition: Mechanical. The sound material in this con-
dition consisted of recordings sourced from a sound design library,
such as bike chains, clockworks, and sounds of various pneumatic
equipment. The robot’s movement changes were accompanied by
clearly accentuated sounds of moving parts becoming aligned with
each other or releasing pressure. During moments, when the robot
did not move, we added ticking noises to represent changes in the
internal mechanics of the machine before a new motion sequence.
We designed this condition to evoke associations of small precise
machinery, such as watches. While we were curious how partici-
pants would perceive the overall sound profile, we expected it to
affect ratings on items relating to movement control and safety.
Session 1: Motion
3.2.3 Sound Condition: Harmonic. The sound material in this con-
dition consists of a low, neutral pitch interval (open fifth), which
is extended to a major chord as the robot approaches the camera.
The arm movement is accompanied by a similar but higher pitched
sound. The harmonies were created using sine waves, resulting in a
sound with reduced brightness, that blended with the other sounds
of the robot. The video additionally includes the sound of Fetch’s
wheels rolling across the wooden floor and the recorded sound of
Fetch moving its arm. The robot’s sound in primarily using neutral
harmonic material as opposed to major or minor chords. This is
based on a recent study by Moore et al., who found that the use
of neutral pitch intervals to represent engine sound was generally
preferred by participants [38]. The robot’s movement changes were
understated on the soundtrack. Changes in Fetch’s direction were
accompanied by fluctuations in the overall pitch of the harmony.
Start and end points of the arm movement were not emphasised,
but the movement sound was faded in and out. We designed this
condition as a possible implementation of blended sonification,
adding harmonic masking elements to an existing motor sound,
and expected it to increase the ratings of some of the measures
when compared to the silent condition.
3.2.4 Sound Condition: Musical. The sound material in this condi-
tion features the same low, neutral pitch interval as the harmonic
condition, albeit with a more pronounced major chord accompany-
ing the movement. It additionally contains a musical interpretation
of the circular motion of turning gears or wheels, represented by a
repeating melodic line that ascends in pitch when the robot moves
forward, and descends in pitch when it turns around its axis. The
robot’s movement changes were accentuated on the soundtrack.
Turning, and the beginning and end of the arm movement were ac-
companied by a high frequency clicking sound. The arm movement
was especially emphasised. It was preceded by a sonic event build-
ing up to the motion, accompanied by bright harmonic elements
which ended in a single note once the movement sequence was
completed. We designed this condition with the aim of replacing
commonly expected robot sound with musical and sound design el-
ements. We were interested in seeing whether this clearly artificial
sound would influence ratings, specifically movement, safety, and
capability. We expected to see reported attractiveness increase.
3.2.5 Sound Condition: Silent. This version was used as a control
and does not feature sound directly emitted by the robot. It does,
however, contain various other sounds in order to make the silence
of the robot convincing. The background soundscape consists of 1)
the sound of Fetch’s wheels quietly rolling over the wooden floor, 2)
the recorded room tone - the ambience of the video recording space
- consisting of noise and occasional clicks from the air conditioning
unit, and 3) sounds created by the person holding the camera, such
as clothes rustling and breathing.
We decided against including an additional sound condition fea-
turing Fetch’s natural sound, because its sonic profile is dominated
by continuous fan noise and while its movement sound is noticeable
when listening to the physical robot, it is much less apparent on
a recording. This can be attributed to the cocktail part effect, the
human ability to selectively focus on sounds in the environment [3].
The most prominent example is the ability to acoustically under-
stand a conversation partner in the loud environment of a cocktail
56
HRI ’21, March 8–11, 2021, Boulder, CO, USA
party. Another example is an orchestra performance where the au-
dience can focus on and thereby better hear the soloists. Listeners
of audio recordings do not have this option, meaning that recorded
sound can lose some of the detail present in a live context. While we
could have accounted for this with microphone placement and mix-
ing techniques, the recording would have become a design process
in itself and the natural sound condition could therefore no longer
be considered an objective representation of the robot sound.
3.2.6 Integration into Video. We integrated the sound of the de-
signed sound conditions into the video with the aim of creating
the impression that all sounds associated with the robot were, in
fact, emitted from the robot and that all sounds heard in the video
were recorded on-site and from the perspective of the camera. To
achieve this, we took various measures. 1) We adjusted the volume
and tone colour of the robot sound to correspond to distance from
the camera. 2) We recorded the room tone on-site and added it to all
sound conditions. 3) We recorded the impulse response of the video
recording space and used it as a reverb for all artificial sound using
convolution [60, Chapter 2]. This essentially made the artificially
added sound reverberate in the physical recording space. All on-site
sounds were recorded using a Zoom H4n Portable Field Recorder.
All sound associated with the robot was mixed to mono and the
room tone and reverb were in stereo. We normalised the integrated
loudness (loudness throughout the entire video) across the mechan-
ical, harmonic and musical sound conditions. The considerably
softer silent condition was not balanced with the others.
3.3 Measures
Participants rated their impressions and expectations of the robot
across four blocks of questions. A first set of questions asked par-
ticipants to rate their agreement with statements such as "I believe
this robot could safely shake my hand" on a 5-point scale ranging
from "strongly disagree" to "strongly agree". A second block asked
participants to rate their impression of the robot along several se-
mantic differential scale items, such as "unattractive-attractive." A
third block presented participants with a list of adjectives, such as
"calm" or "strained", and asked them how well the words described
the robot’s movement on a scale from "not at all" to "extremely
well." The items within each block of questions were randomized,
with the exception of a question regarding whether the video in-
creased the viewer’s stress or anxiety, which was always asked
right after participants had watched the video for the first time. The
items were then averaged into four measures and their consistency
was tested using Cronbach’s alpha. The four measures and their
individual questions are shown in Table 1.
3.3.1 Movement Quality (𝛼 = .82). This measure represents the
participants’ impressions of the robot’s movement. We took items
from Tennent et al.’s "competence" and "aesthetic" measures [48]
and adjusted and extended them to specifically relate to Fetch’s
control over its movement, both from functional (through items
like "uncontrolled") and aesthetic perspectives (through items like
"elegant"). Throughout its movement routine, Fetch performed sud-
den movements and slightly rocked back and forth when coming
to a halt. By either accentuating or distracting from these move-
ments with sound, we expected these movements to be more or
Session 1: Motion
Table 1: Survey Items Making Up the Four Measures
Movement Quality Perceived Safety
smooth This video caused my level of stress or anxiety to increase. (-)
calm This robot would be disturbing in my personal office environment. (-)
uncontrolled (-) I would feel safe leaving this robot in a room with small children.
elegant I would feel comfortable working alongside the robot.
precise I trusted the actions of the robot.
jerky (-) I believe this robot could safely move around people.
strained (-) I believe this robot could safely shake my hand.
less noticed. We also expected ratings for this measure to have
implications for the other measures in the survey.
3.3.2 Perceived Safety (𝛼 = .80). This measure reflects how com-
fortable and safe participants feel with imagining the robot in their
work environment or home. We took items from Tennent et al.’s
"trust" measure [48] and added two additional items, placing the
measure’s focus on safe interactions with people. We expected that
the sound conditions would influence participants’ perceived safety
both positively and negatively. We thought certain sound elements
might unsettle listeners or make them feel at ease and that this
would influence their perception of how safe this robot is to be
around.
3.3.3 Perceived Capability (𝛼 = .86). This measure reflects the range
of abilities and functionality which participants attribute to the
robot after viewing the video clip. We took six items from a robot
capability measure by Cha et al. [8] and hypothesised that designing
the sound of the robot’s internal workings to resemble precise
machinery or heavy industrial equipment could influence what
tasks the robot would be considered capable of.
3.3.4 Attractiveness (𝛼 = .86). Finally, we took the five items of
the "attractiveness" measure by Ho and MacDorman [19] and used
them to measure the participants’ sentiment towards the robot. We
were interested in seeing whether different sound conditions would
generally improve or worsen ratings of the robot, and how these
subjective ratings would relate to the other measures.
3.3.5 Additional Questions. Besides of the four measures above,
we asked three additional questions, shown in Table 2. Question 1
tested whether any of the elements in the sound conditions were
interpreted as deliberate communication addressing the listener.
The hum of the artificial motor in the harmonic condition, for ex-
ample, could potentially be interpreted as a semantic-free utterance
[58]. As the effect of utterances on our measures was not within
the scope of this study, we wanted to ensure none of the sound
conditions showed significant differences. Participants answered
the question on a 5-point scale ranging from "definitely not" to "def-
initely yes". Question 2 asked how relevant participants considered
the robot’s appearance, sound, and movement for their ratings. We
expected there to be no differences in the reported influence of
movement and appearance across the sound conditions, but were
interested in how influential the different sound conditions were
rated. We also expected the absence of sound in the silent con-
dition to be reflected in this metric. Participants answered each
item on a 5-point scale ranging from "not at all" to "a great deal".
Question 3 tested how aware participants were of the presence of
artificial sound. We again expected the silent condition to receive
57
HRI ’21, March 8–11, 2021, Boulder, CO, USA
Perceived Capability Attractiveness
I believe this robot is capable of cleaning up the dining table. Unattractive:Attractive
I believe this robot is capable of organising the pantry. Ugly:Beautiful
I believe this robot is capable of taking out the trash. Repulsive:Agreeable
I believe this robot is capable of bringing a drink. Crude:Stylish
I believe this robot is capable of preparing simple meals. Messy:Sleek
I believe this robot is capable of doing laundry.
the lowest ratings, and wanted to see whether our intention to de-
sign the harmonic condition as realistic was reflected in this rating.
Participants answered the question on a 5-point scale ranging from
"definitely not" to "definitely yes". Additionally, our survey asked
participants to give context to their ratings using written text and
to describe the sound of the robot.
3.4 Study Procedure
We designed a survey2 using the Qualtrics survey software and
recruited participants via the Prolific online platform. This platform
is similar to Amazon Mechanical Turk (AMT), which is widely used
in studies on human-robot interaction and its results have been
reported to match those of laboratory studies well [10, 27, 37]. Pro-
lific provides high-quality data to researchers through its dedicated
focus on research studies [42]. Both AMT and Prolific have been
used in previous studies on the topic of robot sound [38–40, 48].
After signing up for the survey, participants were directed to an
information statement. They were instructed to wear headphones,
but not made aware of the study’s specific focus on sound. Rather,
they were asked to assess the robot’s behaviour. This was done to
avoid participants considering all questions exclusively in relation
to the sound they were hearing, thereby potentially skewing the
results. To test whether participants had functioning audio, they
were asked to type in a spoken number on a test video’s audio
track. Even though more elaborate ways to test for headphone use
are available [57] we considered this type of testing sufficient, as
the sound conditions were considerably different from each other.
After the audio test, participants were randomly assigned to one
of the four sound conditions for the remainder of the survey and
presented with the questions. They were encouraged to revisit the
video if necessary. Towards the end of the survey, participants were
asked whether they had impaired hearing, whether they had used
headphones as requested, and how familiar they were with elec-
tronics, robots, and Fetch in particular. Finally, they were presented
with a debriefing explaining the study’s specific focus on sound and
2 The survey can be found at
https://unsw.au1.qualtrics.com/jfe/form/SV_86Tn7dfp8WOdAyN
Table 2: Additional Questions
1) Do you have the impression the robot tried to communicate something to you?
2) Please indicate how much the following robot characteristics influenced your answers:
The way it looks.
The way it sounds.
The way it moves.
3) Did you have the impression that artificial sound was added to the robot’s motor sound?
Session 1: Motion
given the opportunity to revoke their participation without affect-
ing their payment. The procedure was approved by the institutional
review board of UNSW.
3.5 Participants
Based on a sample size calculation informed by Tennent et al. [48],
we recruited 240 participants (37.5% male, 62.5% female, M =36
years, SD = 11 years) on the Prolific platform. Participants were
nationals of either the United Kingdom or the United States and
spoke English as a first language. They were payed £0.75 for the
task, which was estimated to take six minutes. All participants had
an approval rating of 95% or higher and had taken part in at least
100 studies on the platform. The participants took on average 6.3
minutes to complete the survey, with a minimum of 2.5 minutes
and a maximum of 35. No significant differences in age and prior
experience with electronics, robotics, and Fetch in particular were
reported across sound conditions, and experience with robotics and
Fetch was generally low. One participant revoked their consent
after the debriefing and was removed from the data set prior to
analysis. Three participants incorrectly answered the audio test
question at the beginning of the survey, nine participants reported
impaired hearing and 21 reported not wearing headphones during
the study. These participants were excluded from analysis. The final
group sizes were mechanical: 50, harmonic: 53, musical: 53, and
silent: 50.
4 RESULTS
4.1 Statistical Methods
Statistical analysis was done in R version 4.0.2 [11].3 To arrive at the
measures movement quality, perceived safety, perceived capability,
and attractiveness, we first reversed the ratings of the four ques-
tions describing negative impressions (marked with "(-)" in Table
1). With every item then ranging from 1, negative, to 5, positive,
each measure’s individual items were averaged. The measures’ con-
sistency was calculated using cronbach() from the psy package
[14]. We conducted one-way between-subjects ANOVAs to com-
pare the effect of the four sound conditions on the measures. These
were performed using aov() from the R base package. Where the
ANOVA showed a significant effect (p<.05), we used the Tukey-
HSD() test [1] to adjust p-values for multiple comparisons and see
relationships between the individual sound conditions. Homogene-
ity of variance was tested with Levene’s test using leveneTest()
in the car package [16] and normality was confirmed using the
Shapiro-Wilk test. Bar plots showing means with error bars repre-
senting +-1 standard error were created using ggplot2 [55]. Y axes
have a theoretical range of 1 to 5, but have been limited to improve
readability. Significance levels are highlighted as stars. * indicates
p < .05, ** indicates p < .01, and *** indicates p < .001.
4.2 Measures
4.2.1 Movement Quality. Figure 3 shows the ratings of movement
quality across sound conditions. There was a significant effect of
sound on reported movement quality at the p<.05 level for the four
conditions [F(3, 202) = 11.83, p < 0.001]. Post-hoc comparisons using
3 Survey data and R file are available in the ACM digital library.
58
HRI ’21, March 8–11, 2021, Boulder, CO, USA
the Tukey-HSD test indicated that the mean rating for the musical
condition (M = 3.57, SD = 0.586) was significantly higher than both
the silent condition (M = 3.21, SD = 0.636) and the mechanical
condition (M = 2.82, SD = 0.629). Ratings of the mechanical con-
dition were significantly lower than the harmonic (M = 3.26, SD
= 0.685) and silent condition. The harmonic condition was not
significantly different to the silent and musical condition. Taken
together, compared to the silent robot, the mechanical sound
decreased, the musical sound increased, and the harmonic sound
did not affect movement quality. The effect size was 𝜂 2 = 0.149.
4.2.2 Perceived Safety. Figure 4 shows the ratings of perceived
safety in each sound condition. There was a significant effect of
sound on perceived safety at the p<.05 level for the four condi-
tions [F(3, 202) = 4.048, p = 0.008]. Post-hoc comparisons using the
Tukey-HSD test indicated that the mean rating for the mechanical
condition (M = 2.91, SD = 0.752) was significantly lower than both
the musical condition (M = 3.37, SD = 0.747) and the silent con-
dition (M = 3.35, SD = 0.780), but not significantly lower than the
harmonic condition (M = 3.29, SD = 0.745). The harmonic, musical,
and silent conditions were not significantly different from each
other. In summary, compared to the silent robot, the mechanical
sound decreased perceived safety, while the harmonic and musical
sound conditions did not affect it. The effect size was 𝜂 2 = 0.057.
4.2.3 Perceived Capability. Figure 5 shows the ratings of perceived
capability across the four sound conditions mechanical (M = 2.44,
SD = 0.709), harmonic (M = 2.75, SD = 0.953), musical (M = 2.72, SD
= 0.838), and silent (M = 2.67, SD = 0.914). The ANOVA showed no
significant effect of sound on perceived capability [F(3, 202) = 1.357,
p = 0.257]. None of the individual items making up the perceived
capability measure (see Table 1) significantly differed across sound
conditions either. The effect size was 𝜂 2 = 0.019.
4.2.4 Attractiveness. Figure 6 shows the ratings of attractiveness
in each sound condition. There was a significant effect of sound on
reported attractiveness at the p<.05 level for the four conditions [F(3,
202) = 4.761, p = 0.003]. Post-hoc comparisons using the Tukey-HSD
test indicated that the mean rating for the musical condition (M =
3.46, SD = 0.717) was significantly higher than both the mechanical
(M = 2.99, SD = 0.761) and the harmonic (M = 3.01, SD = 0.780)
conditions. Ratings of the silent condition (M = 3.22, SD = 0.658)
did not significantly differ to any of the three other conditions.
The harmonic and mechanical conditions were not significantly
different either. Taken together, the robot was rated more attractive
when paired with the musical sound than when paired with either
the mechanical or harmonic sound. However, when using the
silent condition as a baseline, none of the sounds significantly
impacted the participant’s ratings of the robot’s attractiveness. The
effect size was 𝜂 2 = 0.066.
4.2.5 Additional Questions. Analysing answers to the questions
shown in Table 2, we found the following results:
1) The reported presence of deliberate communication from the
robot was not significantly different across the sound conditions
[F(3, 202) = 2.243, p = 0.0845], leading us to conclude that no sound
elements were identified as utterances to a significant degree.
2) The reported influence of robot movement on ratings was
not significantly different across the sound conditions [F(3, 202)
Session 1: Motion HRI ’21, March 8–11, 2021, Boulder, CO, USA

Movement Quality Perceived Safety Perceived Capability

4.0 4.0 4.0
*
*
*** *
*
3.5 ** 3.5 3.5

3.0 3.0 3.0

rating

rating

rating
2.5 2.5 2.5

2.0 2.0 2.0

mechanical harmonic musical silent mechanical harmonic musical silent mechanical harmonic musical silent
sound sound sound

Figure 3: Compared to the silent condition, the Figure 4: Perceived safety in the mechanical Figure 5: Participants reported no significant dif-
mechanical sound decreased and the musical sound condition was rated significantly lower than ferences in perceived robot capability across the
sound increased movement quality ratings. in the silent and musical conditions. four sound conditions.

Attractiveness Influence of Sound on Ratings Presence of Artificial Sound

4.0 4.0 4.0
** ***
* *** ***
**
3.5
** 3.5
3.5
*
***
3.0 3.0
rating

rating
3.0
rating

2.5 2.5 2.5

2.0 2.0 2.0

mechanical harmonic musical silent mechanical harmonic musical silent mechanical harmonic musical silent
sound sound sound
Figure 6: The robot was rated more attractive Figure 7: Participants reported no differences in Figure 8: The mechanical and musical condi-
when paired with the musical condition than how much sound affected their impressions across tions were both identified as significantly more ar-
when paired with the mechanical or harmonic the three designed sound conditions. The silent tificial than the harmonic and silent conditions.
sound. condition was rated significantly less impactful.

= 0.825, p = 0.482]. Neither was the reported influence of robot
appearance [F(3, 202) = 1.604, p = 0.19]. There was a significant
effect of sound on the reported influence of sound on the partic-
ipants’ ratings (shown in Figure 7) at the p<.05 level for the four
conditions [F(3, 202) = 6.378, p < 0.001]. Post-hoc comparisons using
the Tukey-HSD test indicated that the mean rating for the silent
condition (M = 2.06, SD = 1.11) was significantly lower than the
mechanical (M = 2.94, SD = 1.35), harmonic (M = 2.81, SD = 1.21),
and musical (M = 3.00, SD = 1.22) conditions. Those three designed
sound conditions showed no significant differences among each
other. In summary, the reported influence of the robot’s movement,
appearance, and sound on participants’ ratings was not affected by
the sound conditions, except in the case of the silent condition,
where the sound was reported as significantly less influential. This
met our expectations, as the silent condition was designed not to
feature any sounds emitted by the robot, despite the wide range of
background sounds present.
3) The presence of artificial sound as reported by participants
is shown in Figure 8. As expected, there was a significant effect at
the p<.05 level for the four conditions [F(3, 202) = 13.72, p < 0.001].
Post-hoc comparisons using the Tukey-HSD test indicated that the
mean ratings for the musical (M = 3.55, SD = 1.20) and mechanical
(M = 3.28, SD = 0.991) conditions were both significantly higher
than those for the harmonic (M = 2.74, SD = 0.964) and silent
(M = 2.4, SD = 0.808) conditions. The mechanical and musical
conditions were not significantly different from each other, and
neither were the harmonic and silent conditions. In summary,
while the sounds in the musical and mechanical conditions were
more likely to be identified as artificial, the ratings for the harmonic
condition did not significantly differ from the silent conditions,
leading us to conclude that the harmonic condition was convincing
as the robot’s natural sound.
5 DISCUSSION
Our various sound conditions both increased and decreased the
participants’ ratings across some of the measures. Two thoroughly
designed and unrealistic sound conditions, mechanical and musi-
cal, showed significant differences to the silent condition. The
most notable differences were in the participants’ ratings of the
robot’s movement, which were both positively and negatively af-
fected. Coupling the same movement with different sounds lead
to the motions being rated as more or less precise, elegant, jerky,
or uncontrolled, among others. However, these impressions of the
robot’s movement had little implications for the more functional
ratings of perceived safety and capability. Looking at these results
in relation to a prior study by Tennent et al. [48], which explored
the effects of servo motor sound on similar measures, we can make
the following comparisons:
1) While the presence of sound generally decreased ratings across
their measures, this was not the case with our artificial sound con-
ditions, as two them, harmonic and musical, were never rated
below the silent condition in any of the measures. This indicates

59
Session 1: Motion
that while the unprocessed sound of servo motor tends to lead to
lower ratings, our artificial sound conditions are at least on par with
silence, both concerning functional measures like perceived capa-
bility and safety, as well as less functional measures like movement
quality and attractiveness.
2) Given that the subtle differences in the servo motor sounds
used by Tennent et al. still resulted in some demonstrated effects,
we expected the broader sonic range of artificial sound conditions
to result in more pronounced differences in ratings. While we found
interesting effects on reported quality of movement, we found little
difference in perceived safety and no differences in perceived capa-
bility. This suggests that sound plays a lesser role in the assessment
of purely functional aspects ("Can this robot prepare a meal?").
When looking at the participants’ comments on the sound, we
found that while descriptions of the mechanical sound were gener-
ally similar ("clicky", "mechanical", etc), descriptions of the musical
condition were not. Participants described the latter as both "sooth-
ing" and "annoying," as "artificial, sinister" and "charming," and as
"pleasant" and "unpleasant." This leads us believe that while bolder
creative approaches to robot movement sound can result in more
pronounced differences in ratings, this introduces the personal taste
of the listener. One could then hypothesise that a careful removal of
potentially controversial elements from the musical condition will
result in higher ratings. However, the harmonic condition, contain-
ing similar harmonic material but none of the more adventurous
elements, was not rated differently to the silent condition in any
of the four measures. We believe this can be addressed by working
towards systems that learn to take personal preference into account,
dynamically adjusting to individuals and context, rather than trying
to please everyone through a common denominator. Viewing the
ratings of the harmonic sound in the context of blended sonification
[52], we found that the subtle (see Figure 8) addition of harmonic
material to the robot’s movement had no significant effect. This
leads us to believe that it is unlikely to affect these measures beyond
masking problematic consequential sound, making it a solution to
a problem which would be better solved by preventing intrusive
motor noise during manufacturing.
While our findings highlight the complexity of the design space
for artificial movement sound, we see them as a promising step
towards an implicit channel of communication that, when combined
with speech or utterances, can provide robotic agents with a more
versatile sonic language. We believe that embodiment- and context-
dependent movement sound has the potential to become a rich and
nuanced modality that may eventually be able to selectively target
specific characteristics, helping designers create more refined and
engaging robot behaviour.
6 LIMITATIONS AND FUTURE WORK
When considering these findings, several limitations should be
noted, especially with regards to real-world implementation. 1)
While online studies help researchers in gathering large data sets
from a broad pool of participants, the limited oversight can intro-
duce noise, which requires stricter quality control. We excluded
data from participants who reported not using headphones, having
impaired hearing, and who incorrectly answered the audio ques-
tion, but we had no control over the models of headphones used
60
HRI ’21, March 8–11, 2021, Boulder, CO, USA
and while the volume across sound conditions was normalised we
could not specify the headphone volume of individual participants.
2) Considering that participants saw a 20-second video clip of the
robot, they had only a small window of opportunity to gather an
impression. While their ratings can give early indicators of how
robots with artificial movement sound are perceived, further studies
involving extended periods of physical or virtual [2] interaction
with the robot are needed to better understand these effects. 3) Prior
HRI studies have found that the impressions caused by robot sound
can be affected by external factors, such as context [44] and embod-
iment [29]. Tennent et al., for example, found that the presence or
absence of a human in their scenario changed participants’ impres-
sions of the robot [48]. 4) Sound localisation [45] and other sound
sources in the environment are also likely to introduce additional
variables in a real world implementation. Given the extensive pos-
sibilities of sound design approaches and the variety of qualitative
comments we received, further work is required to identify which
specific sound elements influence perception across the dimensions
presented in this paper.
7 CONCLUSION
This study gives insight into the effect of artificial movement sound
on robot perception. To our knowledge, it is the first HRI study that
utilises a variety of artificially synthesised sounds to investigate
the influence on people’s perceptions of robot movements, safety,
capability, and attractiveness. We found that our sound conditions
both increased and decreased how participants perceived the robot’s
movement. We further found that despite the fact that movement-
related ratings like "jerky," "precise," and "uncontrolled" differed
significantly across sound conditions, these impressions had little
consequence for the robot’s perceived safety and capability. Sound
conditions that departed further from sounds commonly associated
with robots caused significant differences in ratings, while the more
realistic sound condition did not affect ratings compared to the
silent control. This suggests that musical approaches to robotic
motion sound have the potential to positively impact how people
perceive robotic agents. Subtly harmonic, or entirely non-musical
approaches were found not to be preferable to silence. While the
associations made by participants exposed to the different sound
conditions proved to be hard to predict and, at times, contradictory,
we found that artificial movement sound has the potential to affect
how humans perceive robotic agents. With further study, this sound-
based modality may eventually provide a refined, implicit channel
of communication that enriches interactions and enables robotic
agents to more seamlessly integrate into human environments.
ACKNOWLEDGMENTS
The authors would like to thank Mark Donoghoe (Stats Central,
Mark Wainwright Analytical Centre, UNSW), Katsumi Watanabe,
and Eduardo Benitez Sandoval for their advice on study design and
data analysis, as well as Andrew Haigh for his support during video
and audio recording. The authors also thank the anonymous HRI’21
reviewers and ACs for their valuable and detailed suggestions on
how to strengthen this contribution.
Session 1: Motion
REFERENCES
[1] Hervé Abdi and Lynne J Williams. 2010. Tukey’s honestly significant difference
(HSD) test. Encyclopedia of research design 3 (2010).
[2] Almohannad Albastaki, Marius Hoggenmueller, Frederic Anthony Robinson,
and Luke Hespanhol. 2020. Augmenting Remote Interviews through Virtual
Experience Prototypes. In 32nd Australian Conference on Human-Computer In-
teraction (OzCHI ’20), Sydney, NSW, Australia. ACM, New York, NY, USA, 9.
https://doi.org/10.1145/3441000.3441057 ZSCC: 0000000.
[3] Barry Arons. 1992. A review of the cocktail party effect. Journal of the American
Voice I/O Society 12, 7 (1992).
[4] Jon Bellona, Lin Bai, Luke Dahl, and Amy LaViers. 2017. Empirically Informed
Sound Synthesis Application for Enhancing the Perception of Expressive Robotic
Movement. In Proceedings of the 23rd International Conference on Auditory Display
- ICAD 2017. The International Community for Auditory Display, University Park
Campus. https://doi.org/10.21785/icad2017.049
[5] Bérenger Bramas, Young-Min Kim, and Dong-Soo Kwon. 2008. Design of a sound
system to increase emotional expression impact in human-robot interaction. In
2008 International Conference on Control, Automation and Systems. IEEE.
[6] Cynthia Breazeal, Kerstin Dautenhahn, and Takayuki Kanda. 2016. Social Robot-
ics. In Springer Handbook of Robotics, Bruno Siciliano and Oussama Khatib (Eds.).
Springer International Publishing, Cham, 1935–1972. https://doi.org/10.1007/978-
3-319-32552-1_72
[7] C. Breazeal, C.D. Kidd, A.L. Thomaz, G. Hoffman, and M. Berlin. 2005. Effects of
nonverbal communication on efficiency and robustness in human-robot team-
work. In 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.
IEEE, Edmonton, Alta., Canada. https://doi.org/10.1109/IROS.2005.1545011
[8] Elizabeth Cha, Anca D. Dragan, and Siddhartha S. Srinivasa. 2015. Perceived
robot capability. In 2015 24th IEEE International Symposium on Robot and Human
Interactive Communication (RO-MAN). IEEE, Kobe, Japan, 541–548. https://doi.
org/10.1109/ROMAN.2015.7333656
[9] Elizabeth Cha, Naomi T. Fitter, Yunkyung Kim, Terrence Fong, and Maja J.
Matarić. 2018. Effects of Robot Sound on Auditory Localization in Human-
Robot Collaboration. In Proceedings of the 2018 ACM/IEEE International Con-
ference on Human-Robot Interaction - HRI ’18. ACM Press, Chicago, IL, USA.
https://doi.org/10.1145/3171221.3171285
[10] Kuan-Ta Chen, Chen-Chi Wu, Yu-Chun Chang, and Chin-Laung Lei. 2009. A
crowdsourceable QoE evaluation framework for multimedia content. In Proceed-
ings of the 17th ACM international conference on Multimedia.
[11] R Core Development Team and others. 2016. R: A language and environment for
statistical computing. R foundation for statistical computing Vienna, Austria.
[12] Luke Dahl, Jon Bellona, Lin Bai, and Amy LaViers. 2017. Data-Driven Design of
Sound for Enhancing the Perception of Expressive Robotic Movement. In Pro-
ceedings of the 4th International Conference on Movement Computing - MOCO ’17.
ACM Press, London, United Kingdom. https://doi.org/10.1145/3077981.3078047
[13] Alfred O Effenberg. 2005. Movement sonification: Effects on perception and
action. IEEE multimedia 12, 2 (2005), 53–59.
[14] Bruno Falissard. 2012. psy: Various procedures used in psychometry. R package
version 1 (2012).
[15] Michael Filimowicz. 2020. Foundations in Sound Design for Linear Media: A
Multidisciplinary Approach. Routledge.
[16] John Fox, Gregor Gorjanc Friendly, Spencer Graves, Richard Heiberger, Georges
Monette, Henric Nilsson, Brian Ripley, Sanford Weisberg, Maintainer John Fox,
and MASS Suggests. 2007. The car package. R Foundation for Statistical Computing
(2007).
[17] Emma Frid, Roberto Bresin, and Simon Alexanderson. 2018. Perception of Me-
chanical Sounds Inherent to Expressive Gestures of a NAO Robot-Implications
for Movement Sonification of Humanoids. In Sound and Music Computing.
[18] William Gaver. 2012. What should we expect from research through design?. In
Proceedings of the 2012 ACM annual conference on Human Factors in Computing
Systems - CHI ’12. ACM Press, Austin, Texas, USA. https://doi.org/10.1145/
2207676.2208538
[19] Chin-Chang Ho and Karl F MacDorman. 2010. Revisiting the uncanny valley the-
ory: Developing and validating an alternative to the Godspeed indices. Computers
in Human Behavior 26, 6 (2010).
[20] Guy Hoffman and Wendy Ju. 2014. Designing Robots With Movement in Mind.
Journal of Human-Robot Interaction 3, 1 (March 2014). https://doi.org/10.5898/
JHRI.3.1.Hoffman
[21] Marius Hoggenmueller, Luke Hespanhol, and Martin Tomitsch. 2020. Stop and
Smell the Chalk Flowers: A Robotic Probe for Investigating Urban Interaction
with Physicalised Displays. (2020).
[22] Marius Hoggenmueller, Wen-Ying Lee, Luke Hespanhol, Martin Tomitsch, and
Malte Jung. 2020. Beyond the Robotic Artefact: Capturing Designerly HRI Knowl-
edge through Annotated Portfolios. (2020).
[23] Kaoru Inoue, Kazuyoshi Wada, and Yuko Ito. 2008. Effective application of
Paro: Seal type robots for disabled people in according to ideas of occupational
therapists. In International Conference on Computers for Handicapped Persons.
Springer.
61
HRI ’21, March 8–11, 2021, Boulder, CO, USA
[24] Gunnar Johannsen. 2001. Auditory Displays in Human–Machine Interfaces
of Mobile Robots for Non-Speech Communication with Humans. Journal of
Intelligent and Robotic Systems 32, 2 (2001).
[25] Gunnar Johannsen. 2002. Auditory display of directions and states for mobile
systems. Georgia Institute of Technology.
[26] Gunnar Johannsen. 2004. Auditory displays in human-machine interfaces. Proc.
IEEE 92, 4 (2004).
[27] Patrik Jonell, Taras Kucherenko, Ilaria Torre, and Jonas Beskow. 2020. Can
we trust online crowdworkers? Comparing online and offline participants in a
preference test of virtual agents. arXiv preprint arXiv:2009.10760 (2020).
[28] Veikko Jousmäki and Riitta Hari. 1998. Parchment-skin illusion: sound-biased
touch. Current biology 8, 6 (1998), R190–R191.
[29] Takanori Komatsu and Seiji Yamada. 2011. How Does the Agents’ Appearance
Affect Users’ Interpretation of the Agents’ Attitudes: Experimental Investigation
on Expressing the Same Artificial Sounds From Agents With Different Appear-
ances. International Journal of Human-Computer Interaction 27, 3 (Feb. 2011).
https://doi.org/10.1080/10447318.2011.537209
[30] Gregory Kramer, Bruce Walker, Terri Bonebright, Perry Cook, John H Flowers,
Nadine Miner, and John Neuhoff. 2010. Sonification report: Status of the field
and research agenda. (2010).
[31] Thierry Lageat, Sandor Czellar, and Gilles Laurent. 2003. Engineering hedonic
attributes to generate perceptions of luxury: Consumer perception of an everyday
sound. Marketing Letters 14, 2 (2003).
[32] Adrian Benigno Latupeirissa and Roberto Bresin. 2020. Understanding non-verbal
sound of humanoid robots in films. In Workshop on Mental Models of Robots at
HRI 2020 in Cambridge, UK.
[33] Adrian Benigno Latupeirissa, Panariello Claudio, and Roberto Bresin. 2020. Ex-
ploring emotion perception in sonic HRI. In Sound and Music Computing Confer-
ence, Torino, 24-26 June 2020. Zenodo.
[34] Adrian Benigno Latupeirissa, Emma Frid, and Roberto Bresin. 2019. Sonic char-
acteristics of robots in films. In Sound and Music Computing Conference.
[35] Michal Luria, Samantha Reig, Xiang Zhi Tan, Aaron Steinfeld, Jodi Forlizzi, and
John Zimmerman. 2019. Re-Embodiment and Co-Embodiment: Exploration of
social presence for robots and conversational agents. In Proceedings of the 2019
on Designing Interactive Systems Conference - DIS ’19. ACM Press, San Diego, CA,
USA, 633–644. https://doi.org/10.1145/3322276.3322340
[36] Michal Luria, John Zimmerman, and Jodi Forlizzi. 2019. Championing Research
Through Design in HRI. (2019).
[37] Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on
Amazon’s Mechanical Turk. Behavior research methods 44, 1 (2012).
[38] Dylan Moore, Rebecca Currano, and David Sirkin. 2020. Sound Decisions: How
Synthetic Motor Sounds Improve Autonomous Vehicle-Pedestrian Interactions.
In 12th International Conference on Automotive User Interfaces and Interactive
Vehicular Applications.
[39] Dylan Moore, Tobias Dahl, Paula Varela, Wendy Ju, Tormod Næs, and Ingunn
Berget. 2019. Unintended Consonances: Methods to Understand Robot Motor
Sound Perception. In Proceedings of the 2019 CHI Conference on Human Factors
in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk. https:
//doi.org/10.1145/3290605.3300730
[40] Dylan Moore, Hamish Tennent, Nikolas Martelaro, and Wendy Ju. 2017. Making
Noise Intentional: A Study of Servo Sound Perception. In Proceedings of the 2017
ACM/IEEE International Conference on Human-Robot Interaction - HRI ’17. ACM
Press, Vienna, Austria. https://doi.org/10.1145/2909824.3020238
[41] Ioana Ocnarescu and Isabelle Cossin. 2019. The Contribution of Art and Design
to Robotics. In International Conference on Social Robotics. Springer.
[42] Stefan Palan and Christian Schitter. 2018. Prolific. ac—A subject pool for online
experiments. Journal of Behavioral and Experimental Finance 17 (2018).
[43] Hannah RM Pelikan, Mathias Broth, and Leelo Keevallik. 2020. " Are You Sad,
Cozmo?" How Humans Make Sense of a Home Robot’s Emotion Displays. In
Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Inter-
action.
[44] Robin Read and Tony Belpaeme. 2014. Situational context directs how people
affectively interpret robotic non-linguistic utterances. In Proceedings of the 2014
ACM/IEEE international conference on Human-robot interaction - HRI ’14. ACM
Press, Bielefeld, Germany. https://doi.org/10.1145/2559636.2559680
[45] Frederic Anthony Robinson, Oliver Bown, and Mari Velonaki. 2020. Implicit
Communication through Distributed Sound Design: Exploring a New Modality
in Human-Robot Interaction. In Companion of the 2020 ACM/IEEE International
Conference on Human-Robot Interaction (HRI ’20 Companion). ACM, Cambridge,
United Kingdom. https://doi.org/10.1145/3371382.3377431
[46] Markus Schwenk and Kai O. Arras. 2014. R2-D2 Reloaded: A flexible sound
synthesis system for sonic human-robot interaction design. In The 23rd IEEE
International Symposium on Robot and Human Interactive Communication. IEEE,
Edinburgh, UK. https://doi.org/10.1109/ROMAN.2014.6926247
[47] Charles Spence and Qian Wang. 2015. Sensory expectations elicited by the sounds
of opening the packaging and pouring a beverage. Flavour 4, 1 (Dec. 2015), 35.
https://doi.org/10.1186/s13411-015-0044-y
 Session 1: Motion
[48] Hamish Tennent, Dylan Moore, Malte Jung, and Wendy Ju. 2017. Good vibrations:
How consequential sounds affect perception of robotic arms. In 2017 26th IEEE
International Symposium on Robot and Human Interactive Communication (RO-
MAN). IEEE, Lisbon. https://doi.org/10.1109/ROMAN.2017.8172414
[49] Raquel Thiessen, Daniel J Rea, Diljot S Garcha, Cheng Cheng, and James E Young.
2019. Infrasound for HRI: A Robot Using Low-Frequency Vibrations to Impact
How People Perceive its Actions. In 2019 14th ACM/IEEE International Conference
on Human-Robot Interaction (HRI). IEEE.
[50] Gabriele Trovato, Martin Do, Ömer Terlemez, Christian Mandery, Hiroyuki Ishii,
Nadia Bianchi-Berthouze, Tamim Asfour, and Atsuo Takanishi. 2016. Is hugging a
robot weird? Investigating the influence of robot appearance on users’ perception
of hugging. In 2016 IEEE-RAS 16th International Conference on Humanoid Robots
(Humanoids). IEEE.
[51] Gabriele Trovato, Renato Paredes, Javier Balvin, Francisco Cuellar, Nicolai Baek
Thomsen, Soren Bech, and Zheng-Hua Tan. 2018. The Sound or Silence: Investi-
gating the Influence of Robot Noise on Proxemics. In 2018 27th IEEE International
Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE,
Nanjing. https://doi.org/10.1109/ROMAN.2018.8525795
[52] René Tünnermann, Jan Hammerschmidt, and Thomas Hermann. 2013. Blended
sonification – sonification for casual information interaction. Georgia Institute
of Technology.
[53] René Van Egmond. 2008. The experience of product sounds. In Product experience.
Elsevier.
62
View publication stats
HRI ’21, March 8–11, 2021, Boulder, CO, USA
[54] Katsumi Watanabe and Shinsuke Shimojo. 2001. When Sound Affects Vision:
Effects of Auditory Grouping on Visual Motion Perception. Psychological Science
12, 2 (March 2001), 109–116. https://doi.org/10.1111/1467-9280.00319
[55] Hadley Wickham. 2016. ggplot2: elegant graphics for data analysis. springer.
[56] Jason Wolford, Ben Gabaldon, Jordan Rivas, and Brian Min. 2019. Condition-Based
Robot Audio Techniques. Google Patents.
[57] Kevin J. P. Woods, Max H. Siegel, James Traer, and Josh H. McDermott. 2017.
Headphone screening to facilitate web-based auditory experiments. Attention,
Perception, & Psychophysics 79, 7 (Oct. 2017). https://doi.org/10.3758/s13414-017-
1361-2
[58] Selma Yilmazyildiz, Robin Read, Tony Belpeame, and Werner Verhelst. 2016.
Review of Semantic-Free Utterances in Social Human–Robot Interaction. In-
ternational Journal of Human-Computer Interaction 32, 1 (Jan. 2016). https:
//doi.org/10.1080/10447318.2015.1093856
[59] John Zimmerman, Jodi Forlizzi, and Shelley Evenson. 2007. Research through
design as a method for interaction design research in HCI. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems - CHI ’07. ACM Press,
San Jose, California, USA. https://doi.org/10.1145/1240624.1240704
[60] Udo Zölzer. 2011. DAFX: digital audio effects. John Wiley & Sons. https:
//www.dafx.de/DAFX_Book_Page_2nd_edition/index.html
[61] Elif Özcan and René van Egmond. 2006. Product sound design and application:
An overview. In Proceedings of the fifth international conference on design and
emotion, Gothenburg.