Social Neuroscience

ISSN: 1747-0919 (Print) 1747-0927 (Online) Journal homepage: http://www.tandfonline.com/loi/psns20

The effects of self-involvement on attention,

arousal, and facial expression during
social interaction with virtual others: A
psychophysiological study

Andreas Mojzisch , Leonhard Schilbach , Jens R. Helmert , Sebastian

Pannasch , Boris M. Velichkovsky & Kai Vogeley

To cite this article: Andreas Mojzisch , Leonhard Schilbach , Jens R. Helmert , Sebastian
Pannasch , Boris M. Velichkovsky & Kai Vogeley (2006) The effects of self-involvement
on attention, arousal, and facial expression during social interaction with virtual
others: A psychophysiological study, Social Neuroscience, 1:3-4, 184-195, DOI:
10.1080/17470910600985621

To link to this article: http://dx.doi.org/10.1080/17470910600985621

Published online: 24 Feb 2007.

Submit your article to this journal

Article views: 197

View related articles

Citing articles: 35 View citing articles

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=psns20

Download by: [Universidad Del Norte] Date: 27 September 2015, At: 20:27
 SOCIAL NEUROSCIENCE, 2006, 1 (3
4), 184
195

The effects of self-involvement on attention, arousal, and

facial expression during social interaction with virtual
others: A psychophysiological study

Andreas Mojzisch
Georg-August-University Göttingen, Göttingen, Germany

Leonhard Schilbach
Downloaded by [Universidad Del Norte] at 20:27 27 September 2015

University of Cologne, Cologne, Germany

Jens R. Helmert, Sebastian Pannasch, and Boris M. Velichkovsky

Dresden University of Technology, Dresden, Germany

Kai Vogeley
University of Cologne, Cologne, Germany

Social neuroscience has shed light on the underpinnings of understanding other minds. The current study
investigated the effect of self-involvement during social interaction on attention, arousal, and facial
expression. Specifically, we sought to disentangle the effect of being personally addressed from the effect
of decoding the meaning of another person’s facial expression. To this end, eye movements, pupil size,
and facial electromyographic (EMG) activity were recorded while participants observed virtual
characters gazing at them or looking at someone else. In dynamic animations, the virtual characters
then displayed either socially relevant facial expressions (similar to those used in everyday life situations
to establish interpersonal contact) or arbitrary facial movements. The results show that attention
allocation, as assessed by eye-tracking measurements, was specifically related to self-involvement
regardless of the social meaning being conveyed. Arousal, as measured by pupil size, was primarily
related to perceiving the virtual character’s gender. In contrast, facial EMG activity was determined by
the perception of socially relevant facial expressions irrespective of whom these were directed towards.

INTRODUCTION Woodruff 1978). To illustrate, imagine that while

Understanding other minds plays a pivotal role in
social interaction. It refers to the capacity to
adequately ascribe mental states to others in
order to explain or predict their behavior, an
ability that has been coined ‘‘theory of mind’’ or
‘‘mentalizing’’ (Frith & Frith, 2003; Premack &
having dinner in a restaurant you suddenly
observe an attractive individual who is smiling
at you. Is s/he smiling at you because s/he intends
to flirt with you? Or is s/he just smiling because s/
he is amused by someone else’s joke? And
anyway, is s/he really smiling at you or rather at
the person sitting next to you?

Correspondence should be addressed to: Andreas Mojzisch, Institute of Psychology, Georg-August-University Göttingen,
Gosslerstrasse 14, D-37073 Göttingen, Germany. E-mail: mojzisch@psych.uni-goettingen.de
This work was funded by a grant from the European Commission (NoE COGAIN).
We are grateful to Sven Graupner, Tilman Hensch, Anke Karl, Fiona Mulvey, and Stefan Schulz-Hardt for helpful advice and
support in data analyses.

# 2006 Psychology Press, an imprint of the Taylor & Francis Group, an informa business
www.psypress.com/socialneuroscience DOI:10.1080/17470910600985621

Research integrating approaches from cogni-
tive neuroscience, developmental and experimen-
tal social psychology has recently begun to shed
light on the neural and cognitive mechanisms
underlying our understanding of other minds.
Specifically, distinct regions in the anterior and
posterior temporal cortex and medial prefrontal
cortex have been associated with distinct com-
ponents of social cognition (for reviews see
Amodio & Frith, 2006; Saxe, 2006).
Despite remarkable progress in the neuros-
cientific investigation of social cognition, little
attention has been paid to the effect of personal
involvement during social interaction. Yet it is
one thing to be personally addressed by another
Downloaded by [Universidad Del Norte] at 20:27 27 September 2015
person who intends to communicate with us, but
quite another thing to passively observe another
person intending to communicate with someone
else (Grice, 1975) *also differentiated as theory
of mind on-line and off-line (Frith & Frith, 2003).
Recently, neuroimaging studies have begun to
target aspects of online interactions that require
personal involvement in social interaction (e.g.,
Gallagher, Jack, Roepstorff, & Frith, 2002;
Kampe, Frith, & Frith, 2003; King-Casas, Tomlin,
Anen, Camerer, Quartz, & Montague, 2005;
McCabe, Houser, Ryan, Smith, & Trouard, 2001;
Sanfey, Rilling, Aronson, Nystrom, & Cohen,
2003). However, so far the differential effects of
personal involvement on the neural underpin-
nings of ‘‘mentalizing’’ have not been investigated
systematically.
One of the few exceptions is the study by
Schilbach et al. (2006). In this functional magnetic
resonance imaging (fMRI) study, short animated
video sequences were presented in which partici-
pants were either gazed at by virtual characters or
observed these virtual characters looking at
someone else. In dynamic animations, virtual
characters then either showed socially relevant
facial expressions similar to those used in every-
day life situations to establish interpersonal con-
tact or showed arbitrary facial movements. The
resulting four experimental conditions thus estab-
lished a 2 /2 factorial design. The study revealed
that the ventral part of the medial prefrontal
cortex (vMPFC) was recruited during the decod-
ing of the social meaning being conveyed during
an encounter regardless of whether or not parti-
cipants were personally engaged in social inter-
action. In contrast, a more dorsal part of the
MPFC (dMPFC) was activated during the experi-
ence of self-involvement regardless of whether
the virtual character showed facial expressions
BEING WITH VIRTUAL OTHERS 185
indicative of a communicative intent or arbitrary
facial movements (see Kampe et al., 2003, for
similar findings). This study indicates that distinct
regions of the MPFC are recruited during the
decoding of the social meaning being conveyed
during an encounter and the experience of self-
involvement.
Notwithstanding the progress that has been
made in revealing the neural correlates of under-
standing of other minds, we still know relatively
little about the cognitive operations that are
recruited during self-involvement in social inter-
action or on-line mentalizing. Therefore, the
current study was designed to examine the effect
of self-involvement during social interaction on
attention allocation, arousal, and facial expres-
sion. Eye movements, pupil size, and facial
electromyographic (EMG) activity were recorded
while participants watched the video sequences
developed by Schilbach et al. (2006). Specifically,
we focused on the following questions.
First, we asked whether the allocation of
attention as assessed by eye-tracking measure-
ments would be influenced by being personally
addressed during social interaction. The impor-
tance of allocating attentional resources during
social cognition has been stressed by researchers
from several different disciplines. For example,
mounting evidence from social psychology sug-
gests that the activation of category-based knowl-
edge structures during person perception is
moderated by the availability of attentional
resources (e.g., Gilbert & Hixon, 1991; Quinn &
Macrae, 2005). Another line of research has
focused on individuals with autism who are
severely impaired in understanding other minds.
This research shows that normal adults direct
their attention to the eye region for information
regarding the mental state of others correspond-
ing to the ‘‘language of the eyes’’ (Baron-Cohen,
1995) whereas people with autism have difficul-
ties extracting mental states from the eyes
(Baron-Cohen, Wheelwright, & Jolliffe, 1997).
This is corroborated by eye-tracking studies
suggesting that people with autism spend less
time fixating on the eyes of other persons than
normal adults do (Dalton et al., 2005; Pelphrey,
Sasson, Reznick, Paul, Goldman, & Piven, 2002).
All of these data indicate that the allocation of
attention plays an important role in social cogni-
tion. Yet, the differential effects of being person-
ally addressed during social interaction on
attention allocation and the underlying biological
mechanisms remain unclear.
 186 MOJZISCH ET AL.
Second, we asked whether personal involve-
ment during interaction is associated with an
increase in arousal by measuring pupil diameter.
Notably, studies investigating whether direct eye
gaze increases arousal have reported mixed
results (Gale, Kingsley, Brookes, & Smith, 1978;
Kampe et al., 2003; Nichols & Champness, 1971;
Patterson, Jordan, Hogan, & Frerker, 1981; Strom
& Buck, 1979). The present study aimed to shed
light on this topic by systematically varying
participants’ state of self-involvement.
A third issue that seems to be of crucial
importance in social interaction is whether people
show an adequate emotional reaction upon per-
ceiving another’s communicative intent. EMG
Downloaded by [Universidad Del Norte] at 20:27 27 September 2015
research consistently demonstrates that when
people are exposed to emotional facial expres-
sions, they spontaneously react with distinct facial
EMG reactions. These reactions seem to reflect a
tendency to mimic the interaction partner’s facial
displays (e.g., Bavelas, Black, Lemery, & Mullett,
1986; Dimberg, 1997; Lundqvist, 1995). For ex-
ample, pictures of happy faces spontaneously
evoke increased zygomatic major muscle activity
(i.e., the muscle involved in smiling), whereas
pictures of angry faces evoke increased corruga-
tor supercilii activity (i.e., the muscle involved in
frowning). So far, however, the effects of self-
involvement on facial EMG during social inter-
action have not been examined.
In summary, the purpose of the present study
was to examine attention allocation (eye move-
ments), arousal (pupil size), and facial expres-
sions (facial EMG) of participants while being
personally involved in social interaction as op-
posed to being a passive observer of social
interaction between others. Specifically, we
sought to disentangle the effect of being person-
ally addressed from the effect of decoding the
meaning of another person’s facial expression.
Because electrophysiological studies indicate that
early visual processing is attentive to differences
in the gender of social targets (Mouchetant-
Rostaing & Giard, 2003) and the effects of gender
and mutual gaze appear to interact (Vuilleumier,
George, Lister, Armony, & Driver, 2005), we also
examined the effects of the virtual characters’
gender. Participants were presented with the
video sequences developed by Schilbach et al.
(2006). Within each sequence, a virtual female or
virtual male either turned towards the participant
or looked towards a third person, thereby varying
participants’ state of self-involvement. Virtual
characters then exhibited facial expressions that
were either socially relevant or arbitrary.
METHODS
Participants
The sample included 23 right-handed male parti-
cipants (M /23.4 years, SD /2.4) with no record
of neurological or psychiatric illness. All partici-
pants were naive as to the purpose of the
experiment. They had normal or corrected-to-
normal vision and gave informed consent to
participate in the study.
Design
We extended the 2/2 factorial design of Schil-
bach et al. (2006) by introducing the gender of the
virtual characters as a third independent variable.
As a consequence, the present study was based on
a 2 (Self-involvement: yes vs. no)/2 (Social
Interaction: yes vs. no) /2 (Gender of the virtual
character: female vs. male) within-subjects factor-
ial design.
Dependent variables
To measure the attention allocation, we recorded
participants’ eye movements while they watched
the video sequences developed by Schilbach et al.
(2006). Because eye movements are an overt
manifestation of the dynamic allocation of atten-
tion, they provide information on the operation of
the attentional system. As an index of attention
allocation during scene perception, we measured
fixation duration. Notably, there is a close con-
nection between fixation duration and amount of
attention allocation during information proces-
sing (Inhoff & Radach, 1998; Rayner, 1998;
Velichkovsky, 2001, 2002).
As an index of stimulus-induced arousal, we
measured pupil size. Arousal responses are typi-
cally associated with increased sympathetic activ-
ity, and sympathetic activity induces pupillary
dilatation. Therefore, an increase in arousal is
reflected in an increase in pupil size.
Finally, to measure participants’ facial expres-
sion, we recorded facial EMG activity in the
zygomatic major muscle region.

Apparatus
Eye movements were recorded using the SR
Research Ltd. EyeLink eye-tracking system with
on-line detection of saccades and fixations. Posi-
tion of the eye was detected using bright pupil
image. This head-mounted system has a spatial
resolution of better than 0.5 degrees and a sample
rate of 250 Hz. Fixation onset was detected and
transmitted to the presentation system with a
delay of approximately 12 ms.
Pupil size was measured in arbitrary integer
units referring to pupil diameter. Images were
displayed using a GeForce2 MX card and a CRT
display (19-inch Iiyama Vision Master 451) at
Downloaded by [Universidad Del Norte] at 20:27 27 September 2015
1024 /768 pixels and the frame rate was 100 Hz.
Viewed from a distance of 80 cm, the screen
subtended a visual angle of 278 horizontally and
198 vertically.
EMG activity was measured using bipolar
placement of miniature surface Ag/AgCl electro-
des filled with electrode paste, attached according
to the guidelines by Fridlund and Cacioppo
(1986) at the zygomaticus major muscle sites on
the left side of the face. The participants’ skin was
cleaned before the electrodes were attached.
EMG activity was recorded with a LabLinc
V75
04 Bioamplifier (COULBOURN Instru-
ments, LLC, Allentown, USA), digitized at
1000 Hz and stored on a laboratory computer.
Stimuli
As illustrated in Figure 1, the temporal order of
each video sequence adhered to a standardized
pattern of 7.5 s. Each sequence started with the
entrance of the virtual character up to 1500 ms
(walk in), followed by positioning for 1000 ms
(turn) either towards the participant or towards a
third person (not visible to the participant situ-
ated at an angle of approximately 308). Of
importance for eliciting the target state was the
following time window extending from 2500 ms to
5500 ms (social interaction). This time window
was presented at the same time across all experi-
mental conditions and included the facial expres-
sions or arbitrary movements. Finally, the virtual
character turned away from the participant and
walked out of the screen frame over the last
2000 ms (turn and walk off).
The video sequences were designed using the
software package Poser 4.0 (Curious Lab† ).
BEING WITH VIRTUAL OTHERS 187
Condition-specific changes in face morphology
were obtained by manipulating polygon groups
on a 3D-mesh that made up the virtual character’s
facial structure. The polygon groups were com-
parable to the Action Units as described in the
Facial Action Coding System (FACS; Ekman &
Friesen, 1978). Ten different facial expressions
were used (see Table 1). Pilot studies reported by
Schilbach et al. (2006) showed that five of these
were classified as unambiguously socially relevant
(SOC), that is indicative of the intention to
establish social contact, whereas the other five
were perceived as arbitrary (ARB).
Animation of facial motion was realized by
interpolating images between the neutral and
condition-specific facial expressions as well as
body positions of the virtual characters. AVI-
formatted video sequences were generated by
simulating a 100 mm focal-width camera view.
The video sequences subtended 25.68 horizontally
and 13.38 vertically. As depicted in Figure 2, only
the head and shoulders of the virtual characters
were presented in the video sequences. The
virtual characters appeared against a light grey
background. Across all video sequences, the
following factors were systematically varied and
counterbalanced: appearance on the screen (from
left, from right), direction of gaze (left, right, at
the subject), hair color (four different colors),
hair style (four different styles). All facial expres-
sions or movements were balanced for apparent
motion change. That is, the number of frames that
contained facial motion was balanced with those
that contained static facial configurations.
In total, 100 video sequences were presented
in which 24 different virtual characters exhi-
bited 10 different mimic expressions, 5 of which
were indicative of the intention to establish
interpersonal contact (sample video sequences
are demonstrated at http://www.uk-koeln.de/
kliniken/psychiatrie/Bildgebung/ls_videos/schil-
bach-videos.htm).
Procedure
Participants received standardized instructions on
the computer screen. They were told that they
were part of a virtual 3D scene standing between
two other agents. To help them visualize the
setup, they were shown a picture illustrating this
3D scene (see Figure 3). Addressees to the right
and left side of the participants (OTHER) could
not be seen from the participant’s position (ME).
 188 MOJZISCH ET AL.
Downloaded by [Universidad Del Norte] at 20:27 27 September 2015

Figure 1. Screen shots of exemplary video sequences (ME/virtual character looks at the participant; OTHER/virtual character
looks towards a third person; SOC/virtual character presents a facial expression indicative of a communicative intent; ARB/
virtual character shows an arbitrary facial movement).

After the instruction was given, participants felt that the virtual character’s facial expression
performed a 9-point calibration routine. Calibra- had been indicative of the intention to greet or
tion was repeated if the error in any fixation point initiate contact (CONTACT?). The latter ques-
exceeded 18 or if the average error for all points tion had to be answered by indicating the level of
was above 0.58. Participants first completed two perceived communicative intent on a 4-point
study trials in order to familiarize themselves with scale ranging from 1 (strong feeling of contact)
the task. Calibration was repeated after every to 4 (no feeling of contact). Participants were
second video sequence and each presentation was instructed to respond as soon as they saw the
preceded by a drift correction. signal word on screen.
After each video sequence, participants were
asked to answer two questions each prompted by
a signal word appearing on the screen: first, Artifact control and data reduction
whether the virtual character had looked at the
participant or at the third invisible agent EMG data sets were visually inspected and data
(WHO?); second, whether the participant had sequences with artifacts were excluded from sub-
sequent analysis. EMG data of six participants had
TABLE 1 to be excluded from analyses because of technical
Facial expressions/movements and FACS codes
difficulties and artifacts in the data stream. EMG
Description FACS code signals were highpass filtered (50 Hz), rectified
and integrated with a 30 ms time constant. Areas
Facial expressions Winking 46 under the curve values were calculated for the
(SOC) relevant time windows of each video sequence.
Eyebrow flash 1/2/5B
Smile 12C/25 EMG data were transformed into within-subjects
Eyebrow flash/smile 1/2/5B/12B z-scores prior to statistical analyses.
Smile/winking 12C/46 Owing to technical problems with the eye-
Facial movements ‘‘Ooh’’ (lip speech 22/25 tracking device, one participant had to be ex-
(ARB) configuration) cluded from further analysis of the eye movement
Lip biting 26 data. Fixations shorter than 40 ms as well as
Lip contraction 18
fixations outside of the presentation screen were
Lip thrusting 23
Blow up cheeks AD 33/34 excluded. In sum, 1.6% of all fixations were
discarded by this trimming process.

Downloaded by [Universidad Del Norte] at 20:27 27 September 2015
Figure 2. Two screen shots of exemplary video sequences: (a)
a male virtual character smiling at the participant; (b) a female
virtual character looking towards a third person (not visible to
the participant) and showing an arbitrary facial expression.
Data analyses
We divided the video sequences into three seg-
ments, which were analyzed separately (see also
Figure 1): walk in (400
1500 ms), turn (1500

2500 ms), and social interaction (2500
5500 ms).
Figure 3. Virtual scene shown to participants in the instruction phase to help them visualize the setup.
BEING WITH VIRTUAL OTHERS 189
The start of the walk in segment was defined by
the first frame in which 50% of the head of the
virtual character was visible. For all video se-
quences this condition was fulfilled 400 ms after
the onset.
The rationale for de-composing the video
sequences into three segments was that our
independent variables, by definition, did not
operate equally within each of the three time
segments: In the walk in segment the virtual
character enters the scene. During this part of
the video only the gender of the virtual character
can be discriminated and was hence treated as the
only independent variable in the statistical ana-
lyses. Regarding the turn segment, in which the
virtual character stops and either looks at the
participant or at a third party, both gender and
self-involvement were included in the statistical
analyses. Finally, in the social interaction segment
the virtual character showed either a socially
relevant facial expression or arbitrary facial
movements. Hence, all three independent vari-
ables were included in the statistical analyses.
RESULTS
Fixation duration
For each participant, median fixation duration
was calculated for each of the experimental
conditions and for each of the three time seg-
ments. To analyze the effect of gender on fixation
duration during the walk in segment, the data
 190 MOJZISCH ET AL.
were submitted to a one-way analysis of variance
(ANOVA). The results revealed that virtual
females caused significantly longer fixations
than virtual males, F(1, 21) /21.98, p B/.001,
h2 /.51.
In the turn segment both the gender of the
virtual character and self-involvement were in-
cluded in the analysis. Hence, we submitted the
data to a 2 (Self-involvement: yes vs. no) /2
(Gender: female vs. male) repeated measures
ANOVA. The ANOVA revealed a significant
main effect of Self-involvement, F(1, 21) /6.02,
p /.02, h2 /.22, reflecting that fixation duration
was significantly longer if participants were gazed
at by the virtual character compared to if they
Downloaded by [Universidad Del Norte] at 20:27 27 September 2015
observed the virtual character looking at some-
one else. The effect of Gender and the interaction
of Gender and Self-involvement were not sig-
nificant (both FsB/1).
In the social interaction segment all three
independent variables were included in the ana-
lysis. Therefore, the data were submitted to a 2
(Self-involvement: yes vs. no) /2 (Social Interac-
tion: yes vs. no) /2 (Gender of the virtual
character: female vs. male) repeated measures
ANOVA. The results showed that none of the
main effects were significant (all FsB/1.6, all ps/
.22). There was a trend towards an interaction
between self-involvement and social interaction,
but this trend fell short of significance, F(1, 21) /
3.41, p /.08. See Table 2.
Pupil size
During the walk in segment, the results revealed a
significant effect of Gender indicating that pupil
size was significantly larger if the virtual character
was female than if it was male, F(1, 21) /22.89,
p B/.001, h2 /.52.
During the turn segment, the same effect
emerged, that is, pupil size was significantly larger
for virtual females than for virtual males, F(1,
Mean values of fixation duration (in milliseconds) for the main effects of Self-involvement, Social Interaction, and Gender (standard
errors in parentheses)
Self-involvement
ME OTHER
Walk in 264.34 (13.26) 260.36 (8.47)
Turn 257.50 (10.84) 242.41 (6.37)
Social interaction 279.90 (10.19) 277.93 (7.21)
21) /17.52, p B/.001, h2 /.46. In contrast, Self-
involvement had no significant main effect on
pupil size, F(1, 21) B/1. There was, however, a
significant interaction between Gender and Self-
involvement, F(1, 21) /13.17, p/.002, h2 /.49.
To examine this interaction more closely, simple
effects analyses were conducted. Pupil size was
found to be significantly larger if virtual males
looked directly at the participants rather than if
they looked towards someone beside the partici-
pant, F(1, 21) /8.24, p /.009, h2 /.28. For virtual
females, there was a trend in the opposite direc-
tion, but it fell short of significance, F(1, 21) /
3.31, p/.08, h2 /.14.
During the social interaction segment, the
results again revealed a main effect of gender,
F(1, 21) /71.19, p B/.001, h2 /.77. Moreover,
there was a significant main effect of self-involve-
ment, F(1, 21) /10.46, p /.004, h2 /.33. This
main effect, however, was qualified by a signifi-
cant two-way interaction between gender and
self-involvement, F(1, 21)/13.97, p /.001, h2 /
.40. Simple effects analyses showed that pupil size
was significantly larger if virtual females had
turned to someone beside the participants than
if they looked directly at the participants, F(1,
21) /21.88, p B/.001, h2 /.51. In contrast, for
virtual males no effect of self-involvement
emerged, F(1, 21) B/1. See Table 3.
Facial EMG
According to the literature, a delay of 200 ms to
400 ms in facial muscle responses to a virtual
character’s facial expression can be expected
(Dimberg, 1997, Dimberg & Thunberg, 1998).
Regarding the analysis of the EMG data, we
hence distinguished between the following three
intervals: 600
800 ms (corresponds to the walk in
segment), 2100
2300 ms (corresponds to the turn
segment), and 3500
3700 ms (corresponds to the
social interaction segment).
TABLE 2
Social Interaction Gender
SOC ARB Female Male
258.43 (9.51) 266.27 (10.08) 277.32 (11.55) 247.39 (8.38)
252.23 (9.79) 247.68 (8.80) 249.05 (8.55) 250.86 (9.50)
274.02 (9.53) 283.82 (8.68) 274.55 (7.53) 283.30 (10.17)
 BEING WITH VIRTUAL OTHERS 191

TABLE 3
Mean values of pupil size for the main effects of Self-involvement, Social Interaction, and Gender (standard errors in parentheses)

Self-involvement Social Interaction Gender

ME OTHER SOC ARB Female Male

Walk in 2010.58 (152.31) 2035.10 (158.58) 2005.48 (152.16) 2040.21 (158.74) 2056.47 (155.92) 1989.22 (154.93)
Turn 1947.38 (153.62) 1932.18 (152.57) 1938.23 (149.41) 1941.33 (157.16) 1988.24 (153.62) 1891.32 (149.72)
Social interaction 2043.14 (152.25) 2091.81 (160.51) 2062.88 (153.63) 2072.07 (159.58) 2138.97 (162.14) 1995.98 (150.61)

Of course, we did not expect differential facial
EMG activity during the walk in or the turn
segment because virtual characters did not show
any facial expression until the social interaction
Downloaded by [Universidad Del Norte] at 20:27 27 September 2015
Self-involvement, F(1, 16) /0.84, p /.49, h2 /
.03, indicating that facial expression was deter-
mined by the perception of socially relevant facial
expressions regardless of whom they were direc-

segment. Therefore, EMG effects during the
social interaction segment should be reflected
statistically in significant two-way interactions
between any of the three independent variables
and the time of measurement. We hence sub-
mitted the EMG data to a 2 (Self-involvement:
yes vs. no)/2 (Social Interaction: yes vs. no) /2
(Gender of the virtual character: female vs.
male)/3 (Time of measurement: walk in seg-
ment vs. turn segment vs. social interaction
segment) repeated measures ANOVA.
The only significant effect to emerge from the
statistical analysis was a significant two-way
interaction between Time of measurement and
Social Interaction, F(1, 16) /3.86, p /.03, h2 /
.01. Further analyses revealed that this interaction
effect was due to the fact that there was *
according to our hypothesis *neither an effect
of Social Interaction in the walk in segment, F(1,
16) /0.31, p /.59, h2 /.02, nor in the turn seg-
ment, F(1, 16) /1.53, p /.23, h2 /.09; there was,
however, a significant main effect of Social
Interaction in the social interaction segment,
F(1, 16) /7.66, p/.01, h2 /.32, indicating that
zygomaticus major activity was significantly lar-
ger if the virtual character smiled than if the facial
expression was arbitrary. Importantly, this main
effect of Social Interaction was not qualified by
an interaction between Social Interaction and
ted towards (see Table 4).
DISCUSSION
The present study investigated the effect of self-
involvement during social interaction on atten-
tion, arousal, and facial expression. Specifically,
we aimed to disentangle the effect of being
personally addressed from the effect of decoding
the meaning of another person’s facial expression.
To this end, we recorded eye movements, pupil
size, and facial EMG activity while participants
observed virtual characters gazing at them or
looking towards a third person. The virtual
characters then displayed socially relevant facial
expressions as they would appear in greeting or
approach situations (e.g., smiling) or displayed
arbitrary facial movements.
Our results demonstrate that using anthropo-
morphic virtual characters can be an adequate
tool to study the effects of perception of face-
based cues conveying communicative intent on
measures of attention, arousal and facial expres-
sion. We suggest that this reflects the phenom-
enon of eliciting a sense of ‘‘social presence’’ in
mediated environments (Bailenson, Blascovich,
Beall, & Loomis, 2003; Ijsselsteijn & Riva, 2003).

TABLE 4
Mean values of EMG-activity (z-scores) for the main effects of Self-involvement, Social Interaction, and Gender
(standard errors in parentheses)

Self-involvement Social Interaction Gender

ME OTHER SOC ARB Female Male

Walk in .061 (0.048) /.024 (0.045) .038 (0.050) /.001 (0.041) .006 (0.050) .031 (0.028)
Turn /.007 (0.033) .011 (0.043) /.029 (0.034) .033 (0.033) /.023 (0.041) .027 (0.029)
Social interaction .058 (0.045) /.037 (0.030) .096 (0.059) /.075 (0.021) /.007 (0.028) .028 (0.045)
 192 MOJZISCH ET AL.
Attention
Measures of fixation duration revealed that dur-
ing the initial appearance of the virtual characters
attention allocation was influenced by the gender
of the character. After the virtual character had
turned, however, gender seemingly became less
salient and significant differences of attention
were then determined by whether or not the
face was looking directly at the human observer
regardless of the character’s gender.
This finding nicely ties in with research show-
ing the importance of mutual eye contact for
transmitting information about another person’s
Downloaded by [Universidad Del Norte] at 20:27 27 September 2015
intentions, potentially signaling an upcom-
ing interaction (Argyle & Cook, 1976; Baron-
Cohen, 1995). As Baron-Cohen argued, ‘‘it ma-
kes . . . sense that we should be hypersensitive to
when another organism may be about to attack
us, or may be interested in us for some reason’’
(1995, p. 98). In this sense, it has been proposed
that direct gaze captures the attention of percei-
vers and that this attentional capture increases
the efficiency of cognitive operations (Baron-
Cohen, 1995). Indeed, there is evidence showing
that directed gaze facilitates categorical thinking
(Macrae, Hood, Milne, Rowe, & Mason, 2002)
and improves person memory (Mason, Hood, &
Macrae, 2004). The results of the present study go
beyond this work by demonstrating that being
personally addressed by an animated virtual
character is associated with a significant increase
in the length of fixations. Such an increase in the
length of fixations seems to underlie the pro-
longed analysis of faces with direct gaze (reflected
by longer response times), as has recently been
found by Vuilleumier et al. (2005) for gender
judgments.
Notably, the main effect of attention allocation
observed in our study corresponds to the findings
of Schilbach et al. (2006) of differential neural
activity in dMPFC during the experience of self-
involvement regardless of whether the virtual
character showed socially relevant or arbitrary
facial expressions. Considering the present results
and the results of Schilbach et al. (2006) in
combination, we suggest that dMPFC acts as a
top-down mechanism on the allocation of atten-
tional resources. Accordingly, signals directed at
oneself, such as eye contact, activate the dMPFC,
which, in turn, is the source of top-down signals
that modify the activity in more posterior brain
areas (see Frith & Frith, 2006, for a similar
interpretation).
Interestingly, it has recently been proposed
that dMPFC plays an important role in the
representation of triadic social relations, such as
sharing attention to an object or collaborating on
a shared goal (Saxe, 2006). This form of triadic
attention has to be clearly distinguished from
shared attention as apparent in dyadic engage-
ment (Tomasello, Carpenter, Call, Behne, &
Moll, 2005). Although the present study did not
investigate joint attention per se (i.e., two indivi-
duals attend to the same object because one is
monitoring or following the focus of attention of
the other), it is conceivable that our task estab-
lished a more basic form of triadic attention (i.e.,
I see that the virtual character attends to the
person standing beside me). Notably, a similar
increase in the length of fixations as observed in
the present study has been found in conditions
where states of joint attention have to be main-
tained as a prerequisite for dyadic collaborative
actions (Velichkovsky, 1995). Future studies are
needed that systematically manipulate dyadic as
compared to triadic attention.
Arousal
Throughout the video sequences there was a
significant effect on pupil size dependent upon
the gender of the virtual character. Specifically,
participants showed a larger pupil size if the
virtual character was female than if it was male
(note that all of our participants were male). This
effect was apparent from the first appearance of
the virtual characters in the visual field, that is,
well before the characters’ communicative orien-
tation and their facial expression could be en-
coded. This result is in line with the notion that
our perceptual system is adaptively tuned, main-
taining a chronic vigilance to key features of the
environment that are tied to adaptive challenges
(Gibson, 1979; McArthur & Baron, 1983).
We assume that this constitutes further evi-
dence indicating that participants in our study
did, indeed, react to the anthropomorphic virtual
characters as if they were real people. This also
seems to be reflected in the significant interaction
effect found for gender and self-involvement
during the turn segment: Whereas eye contact
with a male will increase arousal possibly to
prepare for a physiologically charged exchange
of dominance, the absence of eye contact with a

female can be equally distressing. Admittedly,
because only male participants were recruited in
the present study, we cannot be entirely certain
that our results are due to the interaction between
the gender of the participants and the gender of
the virtual characters. Hence, future studies are
required that explore both male and female
participants.
Notably, our results are in line with the results
of Kampe et al. (2003) in showing no main effect
of being personally addressed on stimulus-in-
duced arousal (as measured by pupil size). This
finding provides evidence for the notion that the
effects of direct gaze on information processing *
as observed by Macrae et al. (2002) and Mason
Downloaded by [Universidad Del Norte] at 20:27 27 September 2015
et al. (2004) *are mediated by the selective
allocation of attention and not by an increase in
arousal.
Interestingly, it has been suggested that dimin-
ished duration of eye gaze fixation as observed in
autistic individuals can be explained by the fact
that eye fixation is associated with negatively
valenced over arousal within the autistic group
(Dalton et al., 2005). A promising avenue for
future research would be to investigate whether
the differences in how individuals with autism
scan and process facial information are due to
abnormalities in the neural circuits related to
arousal or rather due to abnormalities in the
neural circuits related to visual attention.
Facial expression
Even newborns tend to imitate facial expressions
and manual gestures (cf. Meltzoff & Moore,
1997). In our study, participants exhibited in-
creased zygomaticus major muscle activity if the
virtual characters showed socially relevant facial
expressions (compared to arbitrary facial move-
ments). Intriguingly, this effect occurred even if
participants were not personally addressed. In
other words, even if participants only observed
the virtual characters smiling at someone else,
they tended to smile back. We hence suggest that
this EMG reaction is subserved by a more basic,
‘‘reflexive’’ mechanism (Satpute & Lieberman,
2006).
Our finding that we tend to mimic other
people even if we are not personally addressed
during social interaction is in line with evidence
suggesting that we mimic facial expressions even
if there is no need to communicate. For example,
it has been shown that viewers tend to mimic the
BEING WITH VIRTUAL OTHERS 193
facial expressions of people on television (e.g.,
Hsee, Hatfield, Carlson, & Chemtob, 1990).
It is particularly interesting to note that the
automatic tendency to imitate the facial expres-
sion of others has been proposed as a means of
helping us to share what other people are
currently feeling. Thus, facial mimicry appears
to bind people together and fosters liking, em-
pathy, and smooth interaction (Decety & Jackson,
2004; Hatfield, Cacioppo, & Rapson, 1994). For
example, in one series of experiments, Van
Baaren, Holland, Kawakami, and Van Knippen-
berg (2004) observed that participants who had
been mimicked were more generous and helpful
toward other people than those participants who
had not been mimicked. Of particular relevance
to the results of our study, Van Baaren et al.
(2004) found that the beneficial consequences of
mimicry were not restricted to behavior directed
toward the mimicker but also included behavior
directed toward individuals not directly involved
in the mimicry situation. Hence, the impact of
mimicry appears to extend beyond the dyad.
Strikingly, the main effect of perceiving so-
cially relevant facial expressions regardless of
self-involvement on facial EMG is in concordance
with the finding of differential activity in vMPFC
for the main effect of perceiving socially relevant
facial expressions (Schilbach et al., 2006). It has
been proposed that vMPFC integrates interocep-
tive, cognitive, and motivational states thereby
representing or producing changes in visceral
sensations as present in affective states (Critchley
et al., 2003). Accordingly, we suggest that vMPFC
involvement can be conceptualized as represent-
ing embodied co-regulative activity between self
and other ‘‘giving us a feel’’ for the meaning of
social signals (see also Saxe, 2006, for a similar
interpretation).
CONCLUSION
Intriguingly, measures of attention allocation,
arousal, and facial muscle activity demonstrate
distinct reaction patterns to different temporal
segments of the stimulus material. We suggest
that these patterns of reactivity subserve and
contribute to the dynamic processes of person
perception by constituting parts of the biological
correlates of our propensity for social perception.
This helps us to actively engage with other
‘‘mindful’’ creatures, to drive the allocation of
attentional resources and to inform higher-order
 194 MOJZISCH ET AL.
cognitive functioning thereby providing essential
information to help us to understand other minds.
REFERENCES
Amodio, D. M., & Frith, C. D. (2006). Meeting of
minds: The medial frontal cortex and social cogni-
tion. Nature Reviews Neuroscience, 7, 268
277.
Argyle, M., & Cook, M. (1976). Gaze and mutual gaze.
Cambridge, UK: Cambridge University Press.
Bailenson, J. N., Blascovich, J., Beall, A. C., & Loomis,
J. M. (2003). Interpersonal distance in immersive
virtual environments. Personality and Social Psy-
Downloaded by [Universidad Del Norte] at 20:27 27 September 2015
chology Bulletin, 29, 1
15.
Baron-Cohen, S. (1995). Mindblindness: An essay on
autism and theory of mind. Cambridge, MA: MIT
Press.
Baron-Cohen, S., Wheelwright, S., & Jolliffe, T. (1997).
Is there a ‘‘language of the eyes’’? Evidence from
normal adults and adults with autism or Asperger
syndrome. Visual Cognition, 4, 311
331.
Bavelas, J. B., Black, A., Lemery, C. R., & Mullett, J.
(1986). ‘‘I show how you feel’’: Motor mimicry as a
communicative act. Journal of Personality and Social
Psychology, 50, 322
329.
Critchley, H. D., Mathias, C. J., Josephs, O., O’Doherty,
J., Zanini, S., Dewar, B. K., et al. (2003). Human
cingulate cortex and autonomic control: Converging
neuroimaging and clinical evidence. Brain, 126,
2139
2152.
Dalton, K. M., Nacewicz, B. M., Johnstone, T., Schae-
fer, H. S., Gernsbacher, M. A., Goldsmith, H. H., et
al. (2005). Gaze fixation and the neural circuitry of
face processing in autism. Nature Neuroscience, 8,
519
526.
Decety, J., & Jackson, P. L. (2004). The functional
architecture of human empathy. Behavioral and
Cognitive Neuroscience Reviews, 3, 71
100.
Dimberg, U. (1997). Facial reactions: Rapidly evoked
emotional responses. Journal of Psychophysiology,
11, 115
123.
Dimberg, U., & Thunberg, M. (1998). Rapid facial
reactions to emotional facial expressions. Scandina-
vian Journal of Psychology, 39, 39
45.
Ekman, P., & Friesen, W. (1978). Facial action coding
system: A technique for the measurement of facial
movement. Palo Alto, CA: Consulting Psychologists
Press.
Fridlund, A. J., & Cacioppo, J. T. (1986). Guidelines for
human electromyographic research. Psychophysiol-
ogy, 23, 567
589.
Frith, C. D., & Frith, U. (2006). How we predict what
other people are going to do. Brain Research, 1079,
36
46.
Frith, U., & Frith, C. D. (2003). Development and
neurophysiology of mentalizing. Philosophical
Transactions of the Royal Society of London, Series
B: Biological Sciences, 358, 459
473.
Gale, A., Kingsley, E., Brookes, S., & Smith, D. (1978).
Cortical arousal and social intimacy in the human
female under different conditions of eye contact.
Behavioral Processes, 3, 271
275.
Gallagher, H. L., Jack, A. I., Roepstorff, A., & Frith, C.
D. (2002). Imaging the intentional stance in a
competitive game. NeuroImage, 16, 814
821.
Gibson, J. J. (1979). The ecological approach to visual
perception. Boston: Houghton Mifflin.
Gilbert, D. T., & Hixon, J. G. (1991). The trouble of
thinking: Activation and application of stereotypic
beliefs. Journal of Personality and Social Psychology,
60, 509
517.
Grice, H. P. (1975). Logic and conversation. In P. Cole
& J. L. Morgan (Eds.), Syntax and semantics. Speech
acts (pp. 41
58). New York: Academic Press.
Hatfield, E., Cacioppo, J. T., & Rapson, R. L. (1994).
Emotional contagion. New York: Cambridge Uni-
versity Press.
Hsee, C. K., Hatfield, E., Carlson, J. G., & Chemtob, C.
(1990). The effect of power on susceptibility to
emotional contagion. Cognition and Emotion, 4,
327
340.
Ijsselsteijn, W. A., & Riva, G. (2003). Being there: The
experience of presence in mediated environments.
In G. Riva, F. Davide, & W. A. Ijsselsteijn (Eds.),
Being there: Concepts, effects and measurement of
user presence in synthetic environments (pp. 1
14).
Amsterdam: IOS Press.
Inhoff, A. W., & Radach, R. (1998). Definition and
computation of oculomotor measures in the study of
cognitive processes. In G. Underwood (Ed.), Eye
guidance in reading and scene perception (pp. 29

54). Oxford, UK: Elsevier.
Kampe, K. K. W., Frith, C. D., & Frith, U. (2003). ‘‘Hey
John!’’: Signals conveying communicative intention
towards the self activate brain regions associated
with mentalizing regardless of modality. Journal of
Neuroscience, 23, 5258
5263.
King-Casas, B., Tomlin, D., Anen, C., Camerer, C. F.,
Quartz, S. R., & Montague, P. R. (2005). Getting to
know you: Reputation and trust in a two-person
economic exchange. Science, 308, 78
83.
Lundqvist, L. O. (1995). Facial EMG reactions to facial
expressions: A case of facial emotional contagion?
Scandinavian Journal of Psychology, 36, 130
141.
Macrae, C. N., Hood, B. M., Milne, A. B., Rowe, A. C.,
& Mason, M. F. (2002). Are you looking at me? Eye
gaze and person perception. Psychological Science,
13, 460
464.
Mason, M. F., Hood, B. M., & Macrae, C. N. (2004).
Look into my eyes: Gaze direction and person
memory. Memory, 12, 637
643.
McArthur, L. Z., & Baron, R. (1983). Toward an
ecological theory of social perception. Psychological
Review, 90, 215
238.
McCabe, K., Houser, D., Ryan, L., Smith, V., &
Trouard, T. (2001). A functional imaging study of
co-operation in two-person reciprocal exchange.
Proceedings of the National Academy of Sciences,
98, 11832
11835.
Meltzoff, A. N., & Moore, M. K. (1997). Explaining
facial imitation: A theoretical model. Early Devel-
opment & Parenting, 6, 179
192.
Mouchetant-Rostaing, Y., & Giard, M. H. (2003).
Electrophysiological correlates of age and gender

perception on human faces. Journal of Cognitive
Neuroscience, 15, 900
910.
Nichols, K., & Champness, B. (1971). Eye gaze and the
GSR. Journal of Experimental Social Psychology, 7,
623
626.
Patterson, M. L., Jordan, A., Hogan, M. B., & Frerker,
D. (1981). Effects of nonverbal intimacy on arousal
and behavioral adjustment. Journal of Nonverbal
Behavior, 5, 184
198.
Pelphrey, K. A., Sasson, N. J., Reznick, J. S., Paul, G.,
Goldman, B. D., & Piven, J. (2002). Visual scanning
of faces in autism. Journal of Autism and Develop-
mental Disorders, 32, 249
261.
Premack, D., & Woodruff, G. (1978). Does the chim-
panzee have a theory of mind? Behavioral and Brain
Sciences, 4, 515
526.
Quinn, K. A., & Macrae, C. N. (2005). Categorizing
others: The dynamics of person construal. Journal of
Downloaded by [Universidad Del Norte] at 20:27 27 September 2015
Personality and Social Psychology, 88, 467
479.
Rayner, K. (1998). Eye movements in reading and
information processing: 20 years of research. Psy-
chological Bulletin, 124, 372
422.
Sanfey, A. G., Rilling, J. K., Aronson, J. A., Nystrom, L.
E., & Cohen, J. D. (2003). The neural basis of
economic decision-making in the ultimatum game.
Science, 300, 1755
1758.
Satpute, A. B., & Lieberman, M. D. (2006). Integrating
automatic and controlled processes into neurocog-
nitive models of social cognition. Brain Research,
1079, 86
97.
Saxe, R. (2006). Uniquely human social cognition.
Current Opinion in Neurobiology, 16, 235
239.
BEING WITH VIRTUAL OTHERS 195
Schilbach, L., Wohlschlaeger, A. M., Kraemer, N. C.,
Newen, A. N., Shah, N. J., Fink, G. R., et al. (2006).
Being with virtual others: Neural correlates of social
interaction. Neuropsychologia, 44, 718
730.
Strom, J., & Buck, R. (1979). Staring and participants’
sex: Physiological and subjective reactions. Person-
ality and Social Psychology Bulletin, 5, 114
117.
Tomasello, M., Carpenter, M., Call, J., Behne, T., &
Moll, H. (2005). Understanding and sharing inten-
tions: The origins of cultural cognition. Behavioral
and Brain Sciences, 28, 675
691.
Van Baaren, R. B., Holland, R. W., Kawakami, K., &
Van Knippenberg, A. (2004). Mimicry and prosocial
behavior. Psychological Science, 15, 71
74.
Velichkovsky, B. M. (1995). Communicating attention:
Gaze position transfer in co-operative problem
solving. Pragmatics and Cognition, 3, 199
222.
Velichkovsky, B. M. (2001). Levels of processing:
Validating the concept. In M. Naveh-Benjamin, M.
Moscovitch, & H. L. Roediger, III (Eds.), Perspec-
tives on human memory and cognitive aging: Essays
in honor of Fergus I. M. Craik. Philadelphia:
Psychology Press.
Velichkovsky, B. M. (2002). Heterarchy of cognition:
The depths and the highs of a framework for
memory research. Memory, 10, 405
419.
Vuilleumier, P., George, N., Lister, V., Armony, J., &
Driver, J. (2005). Effects of perceived mutual gaze
on face judgments and face recognition memory.
Visual Cognition, 12, 85
101.