Sound Emblems for Affective Multimodal Output of a
Robotic Tutor: A Perception Study
Helen Hastie
School of Mathematical and Computer Sciences
Heriot-Watt University
Edinburgh, UK
h.hastie@hw.ac.uk
ABSTRACT
Human and robot tutors alike have to give careful consider-
ation as to how feedback is delivered to students to provide
a motivating yet clear learning context. Here, we performed
a perception study to investigate attitudes towards negative
and positive robot feedback in terms of perceived emotional
valence on the dimensions of ‘Pleasantness’, ‘Politeness’ and
‘Naturalness’. We find that, indeed, negative feedback is
perceived as significantly less polite and pleasant. Unlike
humans who have the capacity to leverage various paralin-
guistic cues to convey subtle variations of meaning and emo-
tional climate, presently robots are much less expressive.
However, they have one advantage that they can combine
synthetic robotic sound emblems with verbal feedback. We
investigate whether these sound emblems, and their posi-
tion in the utterance, can be used to modify the perceived
emotional valence of the robot feedback. We discuss this
in the context of an adaptive robotic tutor interacting with
students in a multimodal learning environment.
CCS Concepts
•Human-centered computing → Natural language in-
terfaces; Auditory feedback; Empirical studies in inter-
action design;
Keywords
Human-robot interaction, multimodal output, speech syn-
thesis, synthesized sounds
1. INTRODUCTION
In the classroom, teachers use multiple ways and modali-
ties to embed knowledge and provide feedback to students.
Similarly, robotic tutors should use a range of modalities
including language, gestures and non-verbal cues. These
non-verbal modalities can be used to emphasize, extend or
even replace the language output produced by a robot. In
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from Permissions@acm.org.
ICMI’16, November 12–16, 2016, Tokyo, Japan
c 2016 ACM. 978-1-4503-4556-9/16/11...$15.00
http://dx.doi.org/10.1145/2993148.2993169
256
Pasquale Dente, Dennis Küster, Arvid
Kappas
Depart. of Psychology and Methods,
Jacobs University, Bremen, Germany
p.dente, d.kuester,
a.kappas@jacobs-university.de
addition, non-verbal cues can make the learning experience
more novel and enjoyable for the students and potentially
help compensate for the uncanny valley effect.
There is much discussion in the literature on the peda-
gogical effectiveness of both negative and positive feedback
[32]. There is, however, a consensus that whatever feedback
is given it needs to be carefully phrased [21], and needs to
be communicated in a clear and unambiguous fashion [13].
The study described here has three stages. The first stage
is to investigate whether negative feedback is perceived to
have less emotional valence in terms of ‘Pleasantness’, ‘Po-
liteness’ and ‘Naturalness’. The second stage is to investi-
gate, if this is the case, could the addition of emotive sound
emblems be used to subtly adjust this valence of the feed-
back. Thirdly, we investigate the effect of positioning the
sound emblems. Sound emblems may be better placed at
the end of an utterance, thus emulating punctuation or they
may be better placed at the beginning of an utterance as the
emblem might be perceived as a spontaneous affect burst,
followed by a verbal elaboration [31].
This work is in the context of an affective robotic or vir-
tual tutor. It has been shown that understanding affect in
a learning environment is key to aiding and assisting stu-
dent progression [22]. While it is understood that too much
emotion is bad for rational thinking, recent findings suggest
that so too is too little emotion as intelligent functioning
is hindered when basic emotions are missing in the brain
[22]. Whilst progress is being made on robots that can ma-
nipulate facial expressions, gestures and language to portray
emotion, some of the more affordable robotic platforms such
as the NAO robot, do have limited facial expressions. In ad-
dition, endowing virtual agents with natural facial emotions
can be computationally expensive and time consuming to
programme. The work discussed here is important to the
field as it gives insights into the use of sound emblems, a
language- and platform-independent mechanism for express-
ing affect in a multimodal system.
2. EMPATHIC ROBOT TUTOR
The robotic tutor was designed as part of the FP7 Emote
project1 to be an empathic tutor who aids students aged
between 11-13 years to exercise their map reading skills as
part of the geography school curriculum. The set-up is mul-
timodal in that a map-related task is presented to them on
a large interactive touch-table or tablet [2]. The students
1
http://www.emote-project.eu
are presented with a series of tasks, which can be solved by
using their skills in reading directions, measuring distances
and identifying symbols on a map. For instance, a sam-
ple task would be “Go North West for 550 meters and find
a museum”. Figure 1 gives the architecture whereby the
robot uses sensors to determine the user’s affect and learner
model, and adapts its interaction accordingly through the
Interaction Manager [8]. The system generates personalised
pedagogical feedback that is aligned with the user’s affect,
for example, the robot gives hints and encouraging remarks
if the user is getting frustrated. Here, we discuss a percep-
tion study that has informed the generation of this affective
feedback as part of the Multimodal Output Manager mod-
ule. All modules and other related resources are available
from the Emote project website.
Figure 1: Architecture of the Emote system
whereby the Multimodal Output Manager is in-
formed by this study
3. PREVIOUS WORK
The ultimate goal of the design of socially intelligent robots
is the development of systems capable of communicating and
interacting with humans at a human-like level of proficiency.
This includes the use of naturally sounding human voices
or other types of sounds to portray emotional feelings [9].
In previous studies, a positive effect has been observed in
robot-child interaction when the NAO robot expresses emo-
tion through voice, posture, whole body poses, eye colour,
and gestures [33]. Other child-robot interaction studies have
indicated that empathy facilitates interaction [2], and that
human-based expressions can be successfully implemented
by robots [18]. Finally, automatic feedback has been closely
examined for tutoring systems [6]. However, little work has
been done on non-human sounds and the effects they have
on interaction, learning, and how the robot is perceived.
Some of the research in this field has relied on sounds
that are rich in music-like qualities, e.g. [9] who produced
a small set of five sounds to represent speech acts. Other
researchers have focused on simpler sounds, such as linear
modifications of pitch or duration of monophonic sounds [14,
16, 15]. Conceptually, much of the research in this field has
been inspired by the sound design of science fiction movies.
Movie portrayals of robots such as R2-D2 and WALL-E are
reflected in recent approaches with regards the creation of
non-verbal sounds by several authors as a means to replace
or augment synthetic speech lacking paralinguistic cues [16,
23, 24]. The perception of abstract sounds, like other per-
ceptual processes in psychology, can be expected to be de-
pendent on further contextual information. As argued by
[23], simple non-linguistic utterances can be used to shift the
burden of interpretation to the human because such sounds
257
have less semantic content than spoken language. Questions,
therefore, arise as to whether verbal utterances with simi-
lar intended valence provide the necessary context to align
with the intended valence of the sound emblem; and ulti-
mately whether sound emblems, when combined with verbal
utterances, can be used to shape the emotional climate of
interaction between robots and humans.
4. METHOD
120 participants (28 female, MAge = 31.02, SDAge = 8.03)
from 35 countries were recruited on Crowdflower’s online
platform2 . Each participant listened to and rated a set of 5
utterances randomly picked from the set of 30 stimuli and
were paid approximately US$1.50 on completion of each set.
A total of approximately 5,500 ratings were collected.
4.1 The Stimuli
The stimuli were composed of either a text-to-speech (TTS)
generated utterance paired with a sound emblem, or a TTS
utterance presented in isolation. Half of the utterances were
negative tutor feedback utterances and half positive feed-
back. Both sets of wording were chosen by learning and
teaching experts and provide varying syllable length and
varying levels of valence, e.g. from high positive valence
“Way to go!” to a more sedate “You got it right”. The TTS
used was the Jack voice, a Received Pronunciation (RP)
male voice from Cereproc3 . The exact wording is given be-
low:
• Positive: “Amazing Job”, “Excellent!”, “Way to go!”,
“Wow!”,“You got it right”
• Negative: “No that’s 15 metres”, “That’s incorrect”,
“OK, so that’s not quite right”, “No that’s a telephone”,
“That solution is wrong unfortunately”
Each of the above-listed 10 utterances were matched up
with a sound emblem with similar intended valence. There
were 3 versions of each of these utterance: one with the em-
blem before the utterance, one with the emblem after and
one with speech only utterance. The sound emblems are dis-
cussed in detail in the following section and were not evalu-
ated in isolation in this study as they have been previously
validated for valence as reported in [11]. Each stimulus was
evaluated on three (7-point) Likert scales, in order to mea-
sure perceived ‘Pleasantness’, ‘Naturalness’, and ‘Politeness’
(Dependent Variables).
‘Pleasantness’ was designed to be used as an estimate of
emotional valence to test if negative robot feedback is per-
ceived as more unpleasant than positive feedback. We de-
cided to investigate ‘Politeness’ as politeness strategies are
commonly employed in one-to-one tutoring interactions and
have been shown to both enhance and inhibit learning [21].
Both ‘Naturalness’ as well as ‘Politeness’ scales have been
used in a number of perceptual tests evaluating the output
of multimodal output modules and natural language gener-
ation systems, in particular to inform surface realisation for
different user personalities [19] and preferred styles [3].
2
http://www.crowdflower.com
3
http://www.cereproc.com
Figure 2: Digital version of the Affect Grid adapted
from [27] as a single-item measure of valence and
arousal. In this example, a judgment (indicated by
the cross) has been made on the 9x9 grid indicating
a response of ‘7’ for valence, and ‘8’ for arousal
4.2 The Emblems
Sound emblems are short nonverbal utterances that are
conceptually separate from vocal “affect bursts” in that they
are on the “pull” end of Scherer’s [29, 30] push-pull dimen-
sion. Scherer derived this conceptual distinction on the basis
of prior work on the acoustic origin of facial expressions and
nonverbal behavior [20, 10, 4], wherein the facial muscula-
ture “pushes” the upper vocal tract to produce “raw affect
bursts”, e.g. laughter [31]. At the other end of this contin-
uum are pull effects that require the shaping of an expression
to conform to socio-cultural models [30]. Vocal emblems,
e.g. “Yuck!” are, therefore, conceptually much closer to ver-
bal interjections than raw affect bursts [31].
In the present research, we studied nonverbal sound em-
blems rather than verbal interjections or affect bursts be-
cause we aimed to interlink these emblems with verbal ut-
terances of a robotic tutor that can be seen as standing in a
cultural tradition of robots such as R2-D2 and others that
have made prominent appearances in movies. These types of
representations have shaped collective cultural expectations
about the sounds to be made by small humanoid-looking
robots. We thus utilize the term sound “emblem” to empha-
size the conceptual distance from raw affect bursts, as well as
to denote the explicit and culturally shaped signaling aims
for which they were originally created.
The sound emblems were selected from BEST, a publicly
available sound database [11]. The BEST sounds were gen-
erated by 9 expert performers on an iPad, via the Bebot app
[1], a touch sound synthesizer. The database has been vali-
dated by 194 unique judges, from 41 different countries. This
evaluation comprised judgments for discrete emotions using
the Differential Emotions Scale [7], as well as judgments for
valence and arousal obtained on the basis of dimensional
valence-arousal circumplex models [25, 26, 34]. To avoid
subject fatigue during these validation studies, sounds were
randomly distributed across 4 subsets, and each individual
sound was evaluated by at least 40 subjects using a digital
version of the 9x9 Affect Grid (see Figure 2) adapted from
[27] to provide a fast and user-friendly single-item measure
for valence and arousal. The Affect Grid is a frequently
used single-item measure [28] that has been independently
258
Figure 3: Sound selection with BEST-IDs of positive
and negative stimuli from the BEST dataset as a
function of their valence and arousal ratings. Error
bars represent standard error of the mean (SEM)
validated and found to be valuable in situations where more
time-consuming measures would not be feasible [12].
Five negative sound emblems, and five positive ones were
selected from the BEST database (see Figure 3) and assigned
randomly to the respective negative and positive TTS utter-
ances. The selection followed two criteria: the duration was
shorter than 2s, and the sound scored high vs. low on va-
lence. After the selection, the sound files (TTS + acoustic
emblems) were equalized and mixed with SoX.
5. RESULTS
We tested three hypotheses for each of the three rating
scales (dimensions): I. (‘Valence’) whether there is indeed
a significant difference between the rating of what we de-
fined ‘positive’ and ‘negative’ stimuli in terms of our three
dimensions of ‘Pleasantness’, ‘Politeness’ and ‘Naturalness’;
II (‘Emblem Addition’) whether adding an emblem reduces
or increases the values of the three dimensions; and III. (‘Po-
sition Factor’) whether the position of the sound emblem,
before or after the utterance, produces a significant differ-
ence on the judges’ attribution. In order to test the above
mentioned hypotheses, we carried out a Repeated-Measures
ANOVA for each of the rating scales (Dependent Variables),
separately. Furthermore, we conducted a series of post-hoc
contrasts. For this purpose, all contrasts were computed
using the Dunnett-Tukey-Kramer (DTK) pairwise multiple
comparison test adjusted for unequal variances and unequal
sample sizes [17].
Figure 4 shows the rating results for the three dimen-
sions. Firstly with regards Hypothesis I, the results con-
firm that for the ‘Valence’, there is a significant difference
(p < 0.05) between the ratings of negative feedback and
positive feedback across two of the three dimensions: ‘Pleas-
antness’ F (1, 119) = 4.92 and ‘Politeness’ F (1, 119) = 4.46.
In other words, positive feedback is considered significantly
more polite and pleasant than negative feedback, regardless
of whether it contains a sound emblem or not. There were
no significant effects for ‘Naturalness’, that is to say neg-
ative and positive stimuli are considered to have the same
‘Naturalness’, which is to be expected given that the TTS
engine and selection of sound emblems are the same across
stimuli.
Figure 4: Mean ratings from a 7 point Likert Scale for ‘Pleasantness’, ‘Politeness’ and ‘Naturalness’. Mean
and standard deviations are given
With regards Hypotheses II and III, for all three dimen-
sions, the ANOVA showed a main effect (p < 0.05) for the
‘Position Factor’ (3 levels: ‘sound before’, ‘sound after’, ‘ut-
terance only’)4 . The DKT post-hoc comparison helps us to
understand that, as it happens, for all three dimensions, the
significant contrasts (p < 0.05) are between ‘utterance only’
vs. ‘sound after’, as well as, between ‘utterance only’ vs.
‘sound before’, but not between ‘sound after’ vs. ‘sound be-
fore’. Therefore, for our participants, the stimuli presented
in the ‘utterance only’ condition are perceived as more pleas-
ant, natural and polite than those stimuli paired with an
utterance with a sound emblem in either position. In other
words, adding an emblem to an utterance (be it positive
or negative) significantly reduces the ‘Pleasantness’, ‘Natu-
ralness’ and ‘Politeness’. For positive feedback, adding an
emblem, in either position, does not increase the valence
(i.e. make it more positive), in fact, surprisingly, it has the
opposite effect and makes it perceived as significantly more
negative on the three dimensions (p < 0.05).
6. DISCUSSION AND FUTURE WORK
Our study suggests that, even if every attempt is made to
make negative feedback sound pleasant and polite, that it
will still be deemed as less so than positive feedback. This
work can, therefore, be used to inform the design of robotic
multimodal output, allowing the system designer to be aware
of the emotional valence induced by the various types of
feedback so as to define strategies to mitigate any negative
consequences. Whilst the results concerning the use of sound
emblems are somewhat unexpected, it may be that the bi-
nary categorisation of negative/positive may be too coarse
and perhaps a more fine-grained and sophisticated pairing
between the utterances is required in terms of utterance con-
tent and length along with the intended valence of the sound
emblems.
In addition, it is possible that users are so unused to hear-
ing sound emblems of this type in everyday conversation that
they were not sure how to judge them. This lack of general
consensus on how to interpret the emblems is reflected by the
4
F (2, 238) = 28.88 for ‘Pleasantness’; F (2, 238) = 22.17 for
‘Politeness’; and F (2, 238) = 32.24 for ‘Naturalness’
259
standard deviations given in Figure 4, which are greater for
those utterances containing a sound emblem. Furthermore,
it should be noted that, while participants in the present
study were informed that they would be listening to robot
speech, they were given no concrete visual representation of
the robot. Therefore, it is possible that basic differences in
perceived ‘Naturalness’ between ‘with sound emblem’ and
‘without sound emblem’ stimuli may have been partially re-
sponsible for the overall reductions in ‘Pleasantness’ for the
composite stimuli. In an embodied presentation situation,
the sounds might thus have been perceived as more congru-
ent with the TTS than it was the case in the present online
study.
One advantage of sound emblems is that they are lan-
guage independent. However, further studies would need to
investigate cultural differences along the lines of the studies
reported in [5]. In addition, future work involves repeating
the study for the target age group of children and with the
physical embodiment of the robot.
In the Emote system for a robotic tutor, the multimodal
output is currently categorised as a binary negative or pos-
itive and the choice of this feedback is triggered by the
inferred affect of the user in terms of four broad affective
states: frustrated, excited, bored and neutral. Future work
would involve making the system adaptive at a more fine-
grained level both in terms of the affect it infers from the
user and the multimodal output it generates. The study de-
scribed here moves the field closer to these highly sensitive
affective systems.
7. ACKNOWLEDGMENTS
This work was supported by the European Commission
(EC) and was funded by the EU FP7 ICT-317923 project
Emote. The authors are solely responsible for the content of
this publication. It does not represent the opinion of the EC,
and the EC is not responsible for any use that might be made
of data appearing therein. We would like to acknowledge the
other Emote partners for their collaboration in developing
the system described here.
8. REFERENCES
[1] R. Black. Bebot - Robot Synth. Normalware, 2014.
[2] G. Castellano, A. Paiva, A. Kappas, R. Aylett,
H. Hastie, W. Barendregt, F. Nabais, and S. Bull.
Towards empathic virtual and robotic tutors. In Proc.
of AIED. 2013.
[3] N. Dethlefs, H. Cuayahuitl, H. Hastie, V. Rieser, and
O. Lemon. Cluster-based prediction of user ratings for
stylistic surface realisation. In Proc. of EACL, 2014.
[4] P. Ekman and W. V. Friesen. The repertoire of
nonverbal behavior: Categories, origins, usage, and
coding. Semiotica, 1(1):49–98, 1969.
[5] H. Elfenbein and N. Ambady. On the universality and
cultural specificity of emotion recognition: a
meta-analysis. 128(2):203–35, 2002.
[6] A. C. Graesser, K. Wiemer-Hastings,
P. Wiemer-Hastings, and R. Kreuz. Autotutor: A
simulation of a human tutor. Cognitive Systems
Research, 1(1):35 – 51, 1999.
[7] C. E. Izard. The psychology of emotions. Springer
Science & Business Media, 1991.
[8] S. Janarthanam, H. Hastie, A. Deshmukh, R. Aylett,
and M. E. Foster. A reusable interaction management
module: Use case for empathic robotic tutoring. In
Proc. of SemDial, 2015.
[9] E.-S. Jee, Y.-J. Jeong, C. H. Kim, and H. Kobayashi.
Sound design for emotion and intention expression of
socially interactive robots. Intelligent Service Robotics,
3(3):199–206, 2010.
[10] H. G. Johnson, P. Ekman, and W. V. Friesen.
Communicative body movements: American emblems.
Semiotica, 15(4):335–354, 1975.
[11] A. Kappas, D. Kuester, P. Dente, and C. Basedow.
Simply the B.E.S.T.! creation and validation of the
bremen emotional sounds toolkit. In Poster presented
at the 1st International Convention of Psychological
Science, Amsterdam, the Netherlands, 2015.
[12] W. Killgore. The affect grid: A moderately valid,
nonspecific measure of pleasure and arousal.
Psychological reports, 83(2):639–642, 1998.
[13] A. N. Kluger and A. DeNisi. The effects of feedback
interventions on performance: A historical review, a
meta-analysis, and a preliminary feedback intervention
theory. 119:254–284, 1996.
[14] T. Komatsu. Can we assign attitudes to a computer
based on its beep sounds? In Proc. of the Affective
Interactions: The computer in the affective loop
Workshop at Intelligent User Interface 2005, 2005.
[15] T. Komatsu, K. Kobayashi, S. Yamada, K. Funakoshi,
and M. Nakano. Augmenting expressivity of artificial
subtle expressions (ASEs): Preliminary design
guideline for ases. In Proc. of the 5th Augmented
Human International Conference, 2014.
[16] T. Komatsu and S. Yamada. How does the agents’
appearance affect users’ interpretation of the agents’
attitudes: Experimental investigation on expressing
the same artificial sounds from agents with different
appearances. Intl. Journal of Human–Computer
Interaction, 27(3):260–279, 2011.
[17] M. K. Lau. DTK: Dunnett-Tukey-Kramer Pairwise
Multiple Comparison Test Adjusted for Unequal
Variances and Unequal Sample Sizes.
260
[18] I. Leite, G. Castellano, A. Pereira, C. Martinho, and
A. Paiva. Modelling empathic behaviour in a robotic
game companion for children: an ethnographic study
in real-world settings. In Proc. of HRI, 2012.
[19] F. Mairesse and M. Walker. Personality generation for
dialogue. In 45th Annual Meeting of the Association
for Computational Linguistics (ACL), 2007.
[20] J. J. Ohala. The acoustic origin of the smile. The
Journal of the Acoustical Society of America,
68(S1):S33–S33, 1980.
[21] N. K. Person, R. J. Kreuz, R. A. Zwaan, and A. C.
Graesser. Pragmatics and pedagogy: Conversational
rules and politeness strategies may inhibit effective
tutoring. Journal of Cognition and Instruction,
13(2):161-188, 1995.
[22] R. W. Picard, S. Papert, W. Bender, B. Blumberg,
C. Breazeal, D. Cavallo, T. Machover, M. Resnick,
D. Roy, and C. Strohecker. Affective learning — a
manifesto. BT Technology Journal, 22(4):253–269,
2002.
[23] R. Read and T. Belpaeme. How to use non-linguistic
utterances to convey emotion in child-robot
interaction. In Proc. of the seventh annual
ACM/IEEE international conference on Human-Robot
Interaction, pages 219–220. ACM, 2012.
[24] R. Read and T. Belpaeme. Situational context directs
how people affectively interpret robotic non-linguistic
utterances. In Proc. of the 2014 ACM/IEEE
international conference on Human-robot interaction,
pages 41–48. ACM, 2014.
[25] J. A. Russell. A circumplex model of affect. J. Pers.
Soc. Psychol., 39(6):1161–1178, 1980.
[26] J. A. Russell. Emotion, core affect, and psychological
construction. Cognition and Emotion,
23(7):1259–1283, 2009.
[27] J. A. Russell, A. Weiss, and G. A. Mendelsohn. Affect
grid: a single-item scale of pleasure and arousal.
Journal of personality and social psychology,
57(3):493, 1989.
[28] Y. I. Russell and F. Gobet. Sinuosity and the affect
grid: A method for adjusting repeated mood scores 1.
Perceptual and motor skills, 114(1):125–136, 2012.
[29] K. R. Scherer. On the symbolic functions of vocal
affect expression. Journal of Language and Social
Psychology, 7(2):79–100, 1988.
[30] K. R. Scherer. Affect bursts. Emotions: Essays on
emotion theory, 161:196, 1994.
[31] M. Schröder. Experimental study of affect bursts.
Speech communication, 40(1):99–116, 2003.
[32] P. Sharp. Behavior modification in the secondary
school: A survey of students’ attitudes to rewards and
praise. 9:109–112, 1985.
[33] M. Tielman, M. Neerincx, J.-J. Meyer, and R. Looije.
Adaptive emotional expression in robot-child
interaction. In Proc. of the 2014 ACM/IEEE
international conference on Human-robot interaction,
pages 407–414. ACM, 2014.
[34] M. Yik, J. A. Russell, and J. H. Steiger. A 12-point
circumplex structure of core affect. Emotion,
11(4):705–731, 2011.