You are on page 1of 10

Social Cognitive and Affective Neuroscience, 2018, 135–144

doi: 10.1093/scan/nsx130
Advance Access Publication Date: 28 October 2017
Original article

Looking at the face and seeing the whole body. Neural
basis of combined face and body expressions
Marta Poyo Solanas,1 Minye Zhan,1 Maarten Vaessen,1 Ruud Hortensius,1,*
Tahnée Engelen,1 and Beatrice de Gelder1,2
Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University,
Limburg 6200 MD, Maastricht, The Netherlands, and 2Department of Computer Science, University College
London, London WC1E 6BT, UK
Correspondence should be addressed to Beatrice de Gelder, Brain and Emotion Laboratory, Department of Cognitive Neuroscience, Faculty of Psychology
and Neuroscience, Maastricht University, Oxfordlaan 55, 6229 EV Maastricht, The Netherlands. E-mail:
*Present address: Wales Institute for Cognitive Neuroscience, School of Psychology, Bangor University, Bangor, Gwynedd, Wales LL57 2AS, UK

In the natural world, faces are not isolated objects but are rather encountered in the context of the whole body. Previous
work has studied the perception of combined faces and bodies using behavioural and electrophysiological measurements,
but the neural correlates of emotional face–body perception still remain unexplored. Here, we combined happy and fearful
faces and bodies to investigate the influence of body expressions on the neural processing of the face, the effect of emo-
tional ambiguity between the two and the role of the amygdala in this process. Our functional magnetic resonance imaging
analyses showed that the activity in motor, prefrontal and visual areas increases when facial expressions are presented to-
gether with bodies rather than in isolation, consistent with the notion that seeing body expressions triggers both emotional
and action-related processes. In contrast, psychophysiological interaction analyses revealed that amygdala modulatory ac-
tivity increases after the presentation of isolated faces when compared to combined faces and bodies. Furthermore, a facial
expression combined with a congruent body enhanced both cortical activity and amygdala functional connectivity when
compared to an incongruent face–body compound. Finally, the results showed that emotional body postures influence the
processing of facial expressions, especially when the emotion conveyed by the body implies danger.

Key words: amygdala; body; emotion; face; fMRI

Introduction as body expressions (Meeren et al., 2005; Van den Stock et al.,
Emotional signalling systems are important regulators of social 2007; Aviezer et al., 2012), emotional voices (de Gelder and
and adaptive behaviour. In this respect, one of the most rele- Vroomen, 2000) and background scenes (Van den Stock and de
vant and studied sources of emotional information is the facial Gelder, 2012; Van den Stock et al., 2014). Here, we specifically in-
expression (Haxby et al., 2000; Adolphs, 2002; Posamentier and vestigate how our perception of facial expressions is influenced
Abdi, 2003; Fusar-Poli et al., 2009). However, although faces have by emotional body postures.
been studied separately for decades, they are not isolated sig- Previous research on the combined perception of faces and
nals but most often seen together with other sources of infor- bodies has so far used behavioural and electroencephalography
mation. Previous studies have shown that the perception of (EEG) measurements. For example, participants are strongly
facial expressions is influenced by various other signals, such biased by the expression of the body when judging facial

Received: 9 June 2017; Revised: 13 September 2017; Accepted: 23 October 2017
C The Author (2017). Published by Oxford University Press.
This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (
licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
For commercial re-use, please contact


Downloaded from
by guest
on 08 January 2018

The study was performed in accordance with the Declaration of Helsinki and all procedures followed the regulations of the Ethical Committee at Maastricht University. Hadjikhani not optimal presentations of the body information but that the and de Gelder.. 1. 2016) and body expression processing (de not a compound and therefore.. Ten presented with different emotional expressions. isolated fearful body (BF).. In addition. Vol. Thus. From left to right.. Van den Stock et al. 2017).e. it and fearful and happy bodies were selected from the Bodily was shown that the information conveyed by the face and the Expressive Action Stimulus Test (de Gelder and Van den Stock. we expected to find higher activity in motor and prefrontal areas when seeing a face shown together with the whole body expres- sion rather than in isolation. We used grey ob- tion of these stimuli. 2014. In this way. it would not serve as a good con- Gelder et al. since bodies not only provide emotional information but also elicit motor preparation (de Gelder et al. Faces were combined with bodies in order to produce ei- (Meeren et al.. 2012. Fearful and happy faces were chosen (CC). isolated happy body (BH). but re- mained unaware of the aim of the study. we hypothesized decreased activity in regions of the frontal midline and motor areas.. isolated fearful face (FF). in situations of ambiguity created by expression mismatch between the face and the body. pound stimuli and preserve the ‘compound’ effect (i. served. 2017). up to bottom: congruent happy face–body compound (CH). temporal trol for the compound between face and body). body is combined at early stages of emotional recognition 2011).. 2017). Based on previous studies of facial and bodily expressions (Meeren et al. Hortensius et al. 2012). de Borst and de control stimulus.8 years. amygdala damage leads to def. 2005). faces and bodies were the processing of emotional face–body compounds. a deactivation of the amygdala and a reduction of cortical activity are observed in comparison to same-emotion face–body pairs. Davis et al. partici- pants either received credit points or were reimbursed with vouchers after their participation in the scan session. nine females. two left-handed participants.g. face included. fearful current study was to elucidate the underlying mechanisms of face with a happy body). incongruent face–body compound with fear- ful body (IF). These results demonstrate the importance ther congruent or incongruent compounds (Meeren et al. the com- and motor areas (Boes et al. The aim of the of a face and a body expressing different emotions (e. isolated happy face (FH). 1 expressions. 2007. All participants had normal or corrected-to-normal vision and a medical history without any psychiatric or neurologic disorders. 2016.. the neural correlates of the perception of combined body) and an incongruent face–body compound was composed faces and bodies still remain largely unexplored. Furthermore. 2014) and alters connectivity with frontal. needs to be pre- sensitive to ambiguity (de Gelder et al. Fig. 1996. 13. 2016. 2005). 2018. a total of nine main conditions and a Gelder (2016) reported that when two faces or two bodies are catch condition comprised the experiment (Figure 1). 2004). congruent fear face–body compound (CF). happy face with a happy To date. 2012) and it is known to be overall outline of the whole body. 2012. In addition.g. decreased activity and modulatory influence of the amygdala would be expected for ambiguous compounds as opposed to unambiguous compounds.. In addition. Finally. 1998.oup. incongruent face–body compound with happy Stimuli body (IH). We also angle replacing the body in the control face stimuli. Downloaded from https://academic. of emotional body expressions in the naturalistic understanding A congruent compound occurred when the combined face and of the role of facial expressions.. Aviezer et al.e. face–body compound control face–body compounds). In fact. Examples of the stimuli of the main conditions employed in the experi- ment (catch condition not included). one of them female) took part in the study. Terburg et al. 2003.... Stimuli consisted of combined face and body expressions ( by guest on 08 January 2018 .. isolated body without head or grey circle replacing the face is Hortensius et al. 1994. even when stimuli are shown briefly (Meeren et al. 136 | Social Cognitive and Affective Neuroscience. This was investigated the modulatory role of the amygdala in the percep. 2005. Participants were informed about the task and the general safety rules of functional magnetic resonance imaging (fMRI) scanning. bination of the grey rectangle and the oval shape was used as Regarding its sensitivity to ambiguous signals. Materials and methods Participants Eighteen healthy participants (mean age ¼ 24.. we also maintain the similarity to the com- Hortensius et al. done for both fearful and happy emotions. body expressed the same emotion (e. No.. an icits in both facial (Adolphs et al. age range ¼ 22–31 years. de Gelder et al.. de Borst and de Gelder. since this area is involved in emotional longs/rectangles because we consider that headless bodies are face and body perception (Morris et al. Hortensius et al. 2005. 2009). Using EEG... from the NimStim Face Stimulus Set (Tottenham et al. the effect of shown separately as control stimuli with a grey oval shape body expressions on the processing of facial expressions and replacing the face in the control body stimuli and a grey rect- the effect of emotional ambiguity between the two.

all the individual anatomical datasets (in Talairach Participants performed a passive oddball task (Sutton et al. a regression model body scanner and a 64-channel head–neck coil (Siemens. screen diagonal ¼ 47 cm) and interpolation were applied to correct for the 3D head mo- situated at the posterior end of the scanner bore. three rest blocks of 10 s each were displayed at specific Netherlands. FA ¼ 9 . 2 mm in-plane isotropic resolution. spatial filtering imately). The scan- folded meshes for a more reliable localization. processing and analysis of the acquired data. multisensory experiments). The functional runs em. posterior direction of encoding. the group analyses were executed and projected on the aver- no overt response was required during the experiment. echo time (TE) ¼ 30 ms. The partici- tion of the participants with respect to the first volume of each pants viewed the stimuli through a mirror attached to the head functional run. The anatomical labelling of the trol for continued attention to the stimuli. interleaved slide acquisition order.. a shape-averaged consecutive and fast presentation of the stimuli. with an inter-stimulus lution structural images for each of the participants (1 mm iso- interval of 200 ms. To improve the spatial correspondence between partici- sess the bias from the unattended to the attended stimulus (i. In order to avoid ticipants using the resulting correspondence information. After alignment. Functional images of the whole brain were obtained using T2*- weighted 2D echo-planar image sequences [number of slices per volume ¼ 64. on the face changed into a red circle for 800 ms (i. the blocks. Furthermore. angles ¼ 2. Participants predictor time courses were convolved with a two-gamma were provided with earplugs to reduce the scanner noise and hemodynamic response function. no stimuli were displayed in order to discarded from the analyses. while ignoring the rest of the body. All foam padding was employed to minimize head movement. together with the gyral/sulcal folding pattern.brainvoyager. matrix size ¼ 256  256. screen height ¼ 24. Moreover. tional data included several steps. Each run consisted of 27 stimulation mm2. This. Germany) located at the Maastricht Brain Imaging conditions and the one corresponding to the oddball block. ning session started when the task had been explained and the participant had understood the instructions. For the group analysis. The pre-processing of the func- counteract any possible adaptation effect. The cortical surfaces were then was used for the pre- time points (after oddball/stimulation block 5.oup. The tor ¼ 2. The images were sized to 354  431 pixels. total screen–eye distance ¼ 75 cm approx- cycles per time course.e. leads us to be- (n ¼ 18) folded cortical mesh was created for both hemispheres. For this purpose. the presen- tation duration of one stimulus within the block). flip angle The experiment consisted of one scan session with two func. Experimental design and task repetition time (TR) ¼ 2000 ms. TE ¼ 2. To con- aged whole-brain folded mesh. M.59 degrees). not that of measuring the ‘spontaneous’ merging of the proach based on individual curvature information reflecting the inputs (e. the Netherlands. participants were resulting clusters was performed according to the atlas of asked after the experiment whether they had always noticed Duvernoy (1999) on individual and averaged group whole-brain the changing shape and colour of the fixation cross. cross located on the face. since we wanted to as- sphere. screen functional data within one volume. A three-dimensional (3D) T1-weighted times per run) included 10 stimuli of the same condition dis.5 cm. TR ¼ 2300 ms. number of volumes per run ¼ 280. tropic resolution. matrix size ¼ 100  100. (FA) ¼ 77 ..98 ms. analysis was performed.e. (MPRAGE) imaging sequence was used to acquire high-reso- played in random order for 800 ms each. functional time series were manually co- rectangle ratio of  1 : 7. was generated consisting of the predictors for each of the nine Erlangen. with a vertical face/oval-body/ After these steps.g. pants’ brains beyond Talairach space. and the was applied to the acquired images with a Gaussian kernel of a stimuli spanned 354  461 pixels on the screen (visual full-width half-maximum of 4 mm. Sinc interpolation was em- Stimuli were displayed using E-Prime 2. lieve that participants did not have enough time to make a con- which were then merged to a whole-brain folded mesh. mirror–eye distance ¼ used to exclude low-frequency drifts in the data of two or fewer 15 cm approximately. Poyo Solanas et al. smoothed cortical functional time series (sampled from 0 to In addition. The oddball blocks were similar to the stimu. 11 and 22). z-transformed Downloaded from https://academic. www.81  3. so every new event 3D Talairach space (2 mm3 resolution) (Talairach and Tournoux. high-pass temporal filtering was coil (screen–mirror distance ¼ 60 cm. participants were asked to pay attention to the 3 mm into grey matter) were subsequently aligned across par- change of the fixation cross into a red circle.V. a random-effects general linear model (GLM) Data were acquired with a 3 T Siemens Magnetom Prisma full. The total dur- ation of a block was 10 s and there was a time interval of 6 s fMRI data pre-processing between blocks. No volumes were During these rest by guest on 08 January 2018 . In addition to the stimulation and oddball BrainVoyager QX (v2.0 software and back- ployed to correct the time difference in slice acquisition of projected onto a screen (screen resolution ¼ 1920  1200.4 Brain Innovation B. multi-band acceleration fac- blocks and 6 oddball blocks presented in random order. total scan time ¼ 6 min 7 s). They were instructed to maintain fixation on a black using a semi-automatic procedure based on intensity values. inflated and The rationale behind the selection of this task is similar to the mapped onto a standard sphere separately for each hemi- paradigms measuring cross-modal bias. The scious interpretation of the compound stimuli. field of view (FoV) ¼ 200  200 ployed a block paradigm. Trilinear/sinc estimation width ¼ 40 cm. total scan time per stimulation blocks (nine distinct categories. 1988). each repeated three run ¼ 9 min 20 s]. no gap. started synchronously with a new scan volume. stimuli were presented in greyscale on a grey background. space) were segmented at the grey–white matter boundary 1965). All contamination of the activation of interest by a motor response. | 137 unique stimuli (five males) were created for each condition. Next.8. The Centre of Maastricht University. Each stimulus presentation was registered with the anatomical images and sinc-interpolated to synchronized to a trigger from the scanner. General linear model analysis of fMRI data fMRI data acquisition At the group level. anterior-to- tional runs and an anatomical run. FoV ¼ 256 lation blocks with the exception that the fixation cross situated  256 mm2. Maastricht. the reconstructed how unattended bodies bias the processing of facial expres- cortices were aligned using a dynamic group averaging ap- sions).

All the predictors were tex. see Figure 2B). ent happy vs congruent fear compounds (CH > CF).05).com/scan/article-abstract/13/1/135/4575135 by guest on 08 January 2018 . the brain present functional coupling with a seed region of whereas in the right hemisphere in MFG. IPS and the right marginal segment included the seed region time course. an increase in activity was found bilat- (A  B). Therefore. 1997. In these comparisons. purpose. Results Psychophysiological interaction analyses Functional activation The activity (see Supplementary Material) and modulatory role of the amygdala were investigated with respect to this experi. 138 | Social Cognitive and Affective Neuroscience. (iii) congruent Eight contrasts were performed to investigate our research fearful vs incongruent compounds with happy bodies (CF > IH). 2018.. ent fearful compounds vs isolated fearful faces (CF > FF). lateral or- avoid having a collinear model. separately. O’Reilly et al. Correction for multiple comparisons involved a similar facial emotions but different body expressions (CF > IH. Particularly. These PPI tional bodies on the processing of facial expressions. the two emotionally incongruent bodies (CH > IF). cingulate cor- predictors and six motion predictors. the original task erally in SFG. 13. increased activity was found in the an- analyses. central sulcus. These included bilateral superior frontal gyrus (SFG).. O’Reilly et al. revealed activation in the left cingulate sulcus.05. precentral gyrus and sulcus. It is import- faces with the same emotional expression were performed ant to note that with this approach the direction of the (IH > FF. the differences between con. medial orbito-frontal interest (physical component. MFG. Vuilleumier et al. All the pre.. alpha level ¼ 0. 2012). To that end. no significant activation was 1997. (ii) the task contrast of interest (psychological clusters in the middle occipital gyrus (MOG). To understand the involvement of the amygdala. inferior temporal component). cipital gyrus (IOG) and ITG (Figure 2D). cific contrast (Friston et al. also included in the model (O’Reilly et al. Functional connectivity between the amygdala and other brain postcentral gyrus and sulcus and superior parietal gyrus (SPG). (vi) incongruent compounds with face–body conditions. This type of analysis aims to identify which voxels in terior middle frontal gyrus (MFG) and lateral orbital gyrus. (v) incongruent compounds with happy bodies vs face–body conditions were compared with the two congruent isolated fearful faces (IH > FF). In contrast. (vii) congruent gruent fear and congruent happiness were explored (CH > CF). and congruent happy compounds vs incongruent compounds gated by contrasting two conditions or two groups of conditions with fearful bodies (CH > IF). PPI analysis: (i) congruent vs incongruent compounds. 2012). erations. Also. MFG. Whereas the latter contrast only (A  B). This interaction Our second goal was to explore the effect of emotional term was obtained by the element-by-element product of the bodies on the processing of facial expressions.. precentral gyrus and sulcus. gyrus and sulcus and SPG. congruent fear compounds as opposed to incongru- dictors of our original GLM that were not involved in generating ent compounds with happy bodies showed significant increase the PPI predictor were also included in the design matrix to in response in left inferior frontal sulcus (IFS). No. the PPI model bital gyrus. The time course of the seed region was ob.. Also. gyrus (ITG) and IPS of the left hemisphere responded more tween the task and the time course of the seed region (PPI pre.oup. as only one task predictor can be used to generate the PPI evoked an activation pattern that resembled the one observed predictor. the contrast of interest of cingulate sulcus. Peelen et al. CH > FH.. CF > FF). the pounds vs incongruent compounds with happy bodies (CF > IH) interaction between the amygdala and other areas was investi. 2015). For this pur- processing of emotional face and body expressions (Morris et al. tained by defining the left and right amygdalae for every partici. 2007. 2004. initial P ¼ 0. In addition. Congruent as opposed to in- 2007. fearful bodies vs isolated happy faces (IF > FH). Vuilleumier and Driver. in a consistent manner (see Supplementary Material). inferior oc- Therefore. de Gelder et al.05). gruent compounds. (viii) congru- The third aim of the study was to elucidate the effect of emo. pose. 2012). dictor) (Friston et al. (ii) congru- ance predictors.. O’Reilly et al. postcentral convolved with the canonical hemodynamic response function. (iii) a predictor representing the interaction be. (CH þ CF > IH þIF. the former comparison lyses. see Figure 2A). the same Four further contrasts looked at the effect of bodily expres- contrasts performed for the functional analyses were used for the sions on the processing of the face. 1 motion predictors were incorporated into the model as nuis. For this analyses were performed for left and right amygdala. questions. instead of just contrasting one condition to baseline. central sulcus. In order to examine the emotional congruency effect (iv) congruent happy vs incongruent compounds with fearful between faces and bodies. alpha level ¼ 0. When the two congruent conditions were sign matrix: (i) the time course of the seed region (physiological compared with each other (CH > CF. For that purpose.. new task predictors were created for the PPI ana. only three component). All contrasts were corrected information flow between areas cannot be interpreted. were compared with the two incongruent compound conditions Vuilleumier. superior frontal sulcus (SFS). in our study. cingulate sulcus. intraparietal sulcus (IPS) and the marginal segment of given context or task (psychological component) (Friston et al. strongly to happy congruent compounds. IF > FH.. 2005. The first goal of this study was to examine the neural correlates mental design given the involvement of this structure in the of emotional congruency in face–body compounds. happy compounds vs isolated happy faces (CH > FH). areas was explored with psychophysiological interaction (PPI) In the left hemisphere.. In addition. the A þ B predictor.. two contrasts compared face–body compounds with at the group level. the PPI predictor. Vol. For carrying out the PPI analysis. four contrasts comparing compounds to isolated tion (5000 iterations. the two emotionally congruent compound conditions 1998. 1997. (demeaned) seed region time course and the (mean-centred) compounds that presented the same emotion in the face but task time course. the right and left congruent compounds enhanced the activity of a large number amygdala were defined as regions of interest for all participants of areas. Downloaded from https://academic. cingulate cortex. 2003. Hadjikhani and de Gelder. MOG. but only for multiple comparisons on the surface with a cluster-level that there is a change in covariation between them for that spe- threshold procedure based on Monte Carlo simulation (5000 it. three observed for incongruent compounds when compared to con- specific variable columns (predictors) were prepared in the de. cluster-level threshold procedure based on Monte Carlo simula- CH > IF). This yielded two different comparisons: congruent fear com- pant as previously explained in this paper. an A þ B predictor was in the congruent vs incongruent compound contrast (Figure 2C). different bodily expression were compared with each other. initial P ¼ 0.05. To control for shared variance. as well as MFG. 2012). the amygdala) for a gyrus.

calcarine sulcus. | 139 Fig. cuneus. (E) IH > FF: incon- gruent compounds with happy bodies vs isolated fearful faces. When compar. significant increase in BOLD re- ing incongruent compounds with fearful bodies with isolated sponse occurred for the congruent fear compounds in bilateral happy faces (IF > FH. MOG. while pared congruent fear compounds with isolated fearful faces in the right hemisphere higher activity was observed in gyrus (CF > FF. gyrus descendens. fusiform gyrus (FG). gyrus descendens. 2. Also. (H) CF > FF: congruent fear compounds vs isolated fearful faces. (A) The two congruent face–body compound conditions (CH þ CF) are compared with the two incongruent ones (IF þ IH). initial P-value of 0. left SFG and left isthmus. (B) CH > CF: congruent happy compounds vs congruent fearful compounds. dens. compounds were contrasted with isolated faces that presented contrast showed two clusters in the left cingulate sulcus and matching emotional expressions to the faces in the compounds. M. some clusters in the left hemisphere carine sulcus. LG. FG. see Figure 2F). Bilaterally. cuneus. isolated happy faces (CH > FH. Poyo Solanas et al. (F) IF > FH: incongruent compounds with fearful bodies vs isolated happy faces. gyrus descendens. Results of the group functional activation analyses (cluster size corrected. MOG. significantly increased activity occurred in the congruent happy compounds. In the left MOG. parieto-occipital incisure. see Figure 2H). FG. SFS and the superior part of precentral sul- gyrus (IFG). (D) CH > IF: congruent happy compounds vs incongruent compounds with fearful bodies. Although no brain regions showed pref- rectus. enhanced activity was re. IOG. inferior frontal vated the left SFG. cal. (C) CF > IH: congruent fear compounds vs incongruent compounds with happy bodies. The first comparison yielded more activity for incongruent com. see Figure 2G) significantly acti- see Figure 2E) in bilateral medial orbital gyrus. MFG. higher activity was also observed in ITG. cuneus and LG for hemisphere. ITG.05). The comparison between congruent happy compounds and pounds with happy bodies as opposed to fearful faces (IH > FF. The last of these contrasts com- medial frontopolar gyrus. Downloaded from https://academic. cus. vealed in bilateral ITG. IOG. (G) CH > FH: congruent happy compounds vs isolated happy faces. calcarine by guest on 08 January 2018 . The opposite were found in the superior and medial frontopolar gyrus.oup. LG. lingual gyrus (LG) and cuneus. precentral gyrus and isthmus. erence for the fearful faces. IOG. MFG and superior temporal sulcus (STS). isthmus and FG. MOG. calcarine sulcus. ITG. gyrus descen.

right temporal pole. bodies with the left posterior cingulate gyrus. tional coupling of the amygdala with other areas was enhanced dala showed increased functional interaction with the right for isolated faces in comparison to face–body compounds. together with bodies rather than in isolation. gyrus comparison between congruent and incongruent face–body rectus. allowing an enhance- dala and left IPS and STG. somatosensory cortex. increased functional coupling was found between the right pounds with fearful bodies and isolated happy faces (IF > FH. 13. Other ITG for congruent fear compounds. body posture enhanced cortical activity and the modulatory The third PPI analysis yielded higher functional connectivity activity of the amygdala when compared to an incongruent between the left amygdala and left insula for congruent fear face–body compound. 2016). LG. Right visual field in happy bodies than in fearful bodies. given the intrinsic properties of the fMRI data. When comparing congru- ent fear compounds with isolated fearful faces (CF > FF. like the amygdala. left MTG. No. see PPI analyses Figure 3H). For the opposite contrast. func- ent happy compounds. When Discussion contrasting the two congruent compound conditions with each We examined the neural correlates of the perception of emo- other (CH > CF. precentral gyrus. greater functional interaction for the incongruent com. left dala as seed regions to investigate its task-dependent inter. right anterior cingulate sulcus and gyrus. see Figure 3B). amygdala and left dorsal ACC and medial part of SFG for con- see Figure 3F) revealed higher coupling for the isolated face con. also for congruent fear compounds as opposed to happy also found that an emotional face combined with a congruent compounds. de Borst and de Gelder. found between right amygdala and other areas. of the functional analysis. All these areas displayed more correlated visual areas increase their activity when faces were presented activity with the seed region for congruent fear than for congru. the IPS. no def- paracentral lobule. action between left amygdala and anterior cingulate gyrus. ventromedial pre- frontal cortex (vmPFC) and medial orbito-frontal cortex (mOFC) involvement of the amygdala showed more correlated activity with left amygdala. Consistent with 3D). see Figure 3C). whereas increased coupling brain areas also showed higher activity for unambiguous com- for incongruent compounds with happy bodies was found with pounds. 2017). postcentral sulcus and anterior cingulate sulcus.. This specific increase in correlated tory nature of these processes or their directionality. pos. of the amygdala when incongruent compounds with happy consequently reducing their modulatory influence over cortical bodies were compared with isolated fearful faces (IH > FF. for ambiguous face–body compounds. This was indeed the case for the results MTS. pounds was observed between the left amygdala and right SFG. 2008. terior cingulate gyrus. activity was also observed between left amygdala and left Additionally. see Figure dorsolateral prefrontal cortex (DLPFC) and OFC. left Effect of congruency in face–body compounds and STS and right anterior cingulate cortex (ACC). higher The PPI analysis contrasting congruent happy compounds response was found for congruent happy compounds in pri- with isolated happy faces (CH > FH. gruent as opposed to incongruent compounds (Figure 3A). paracentral lobule. central sul. previous work (Rudrauf et al. compounds as opposed to incongruent compounds with happy bodies (CF > IH. right SFG. For the same PPI analysis. We SFG. no significant functional interaction was found between the left amygdala and other brain areas. ACC. For the same comparison. see Figure 3G) yielded mary and associative visual cortices in comparison to congruent higher functional interaction between the left amygdala and fearful compounds (de Borst and de Gelder. Congruent fear compounds only elicited higher functional inter- ial part of SFG. The comparison between incongruent com. as previously suggested in the literature (Rudrauf related activity for the incongruent compounds with fearful et al. frontal and temporal gyrus (STG). Hortensius et al. right angular gyrus. cingulate sulcus and the right amygdala for con. right SFG. dala in the perception of combined face and body expressions. (CH > CF. Increased connectivity was found between left amyg.. Janak and Tye. superior parietal lobule (SPL). vmPFC and mOFC for the isolated fearful face condition. These included regions of the primary and associative vmPFC. dition between right amygdala and left subgenual cingulate. gruent stimuli as opposed to incongruent (Figure 3A). no brain areas exhibited higher activity insula. ment of brain activity. 1 cingulate gyrus and inferior precentral gyrus and sulcus. see areas. On the contrary. left However. the left amyg. In contrast. due to the fact that the arms were more extended across the cus. 2016). unambiguous emotional information Figure 3E). Vol. Figure 2B).oup. One possible explanation For the same contrast. for this effect. calcarine sulcus and parieto-occipital incisure for the by guest on 08 January 2018 . egory on the processing of congruent face–body compounds dala and SFG increased for isolated happy faces. congruent compounds with fearful bodies (CH > IF. IOG and left anterior superior Functional activation analyses revealed that motor. MTG and the superior part of the postcentral sulcus. This may be left medial orbital gyrus. In line with the existing literature. posterior cingulate sulcus. right gyrus rectus and inite conclusions can be drawn about the excitatory or inhibi- right medial orbital gyrus. activating Downloaded from https://academic.. amygdala presented higher functional coupling with bilateral actions with other brain areas. whereas the coupling between left amyg. 2018. 140 | Social Cognitive and Affective Neuroscience. 2015. higher activity was observed in comparison for the right amygdala revealed higher task. no significant task-dependent connectivity was Ten PPI analyses were performed with the right and left amyg. task-dependent coupling was tional face–body compounds and the involvement of the amyg- observed between the right amygdala and bilateral FG. we explored the effect of each emotion cat- cuneus and right STS. STS. amygdala also displayed increased functional connectivity for whereas in the right hemisphere in STS. middle temporal gyrus congruent happy compounds as opposed to isolated faces with (MTG) and angular gyrus. left IOG and gyrus descendens. 2016. However. 2008. could be that conflicting information leads to concurrent recip- The next PPI analysis examined the functional connectivity rocal inhibition of subcortical structures. motor areas in response to emotionally congruent face–body dependent connectivity with middle temporal sulcus (MTS) and compounds as opposed to incongruent ones (Figure 2A). would not elicit this mutual inhibition. compounds revealed increased coupling between the left med. and between right amygdala and ITG. since lated fearful faces. right amygdala also showed stronger cor. but also of the PPI analysis. de Borst and de Gelder. When comparing congruent happy compounds with in. The same In line with our hypothesis. In the first PPI analysis.

(E) IH > FF: incongruent compounds with happy bodies vs isolated fearful by guest on 08 January 2018 . Poyo Solanas et al. M. 3. (G) CH > FH: congruent happy compounds vs isolated happy faces.oup. (A) The two congruent face–body compound conditions (CH þ CF) are compared with the two incongruent ones (IF þ IH). | 141 Fig. Results of the group PPI analyses for both the right and left amygdalae (cluster size corrected. initial P-value of 0. (D) CH > IF: congruent happy compounds vs incongruent compounds with fearful bodies.05). (H) CF > FF: congruent fear compounds vs isolated fearful faces. (F) IF > FH: incongruent compounds with fearful bodies vs isolated happy faces. (B) CH > CF: congruent happy compounds vs congruent fearful compounds. (C) CF > IH: con- gruent fear compounds vs incongruent compounds with happy bodies. Downloaded from https://academic.

2006). the amygdala showed higher Downing.. interaction faces. could control for an adequate with matching facial expressions but different bodily emotions response to the given situation. Supplementary Figure S1B).. these by guest on 08 January 2018 . tion (Wolpert et al.. motor preparation (preSMA). emotion and executive structures after the pres. 2006. In the case of congruent significantly more than with a happy body. in supports the notion that body postures not only provide emo- charge of the elaboration of a motor plan in response to that tional and social information but also emotion-related action Downloaded from https://academic. 2002. 1998.. the addition of a fearful body to a happy face also triggered the tional information of the face and the body. indicate that observed from the comparison between two compounds that when faces and bodies are presented together. even when motor... whereas iso- motor area (preSMA). emotional and social processing (anterior temporal lobe and PCC) (Wang Effect of emotional bodies on the processing of facial et al. but also emotion-related action intentions (de Gelder frontal and visual areas respond more strongly when faces are et al. of both the amygdala and cortical structures more strongly. 1998. 2007. A similar result was also These findings. determining whether this information is further pro. Interestingly. Vuilleumier. Peelen and Regarding the PPI analyses. de Borst and de Gelder. Peelen and Downing. 2017). Pichon et al. Figure 2C). our results suggest that somatosensory.. ity between the amygdala and areas related to the monitoring sensory. 2004). 13. 2016). 2005.. Wang et al. ation with a fearful body recruited motor and prefrontal areas 2008. like visual informa. this structure is also between combined faces and bodies. This Yet this effect of fear was not observed in the case of isolated suggests that in the presence of a clear threat signal. 2005). since the fa- potential threat (i. increased the functional sponses in posterior STS. 2009). DLPFC and coupling between the right amygdala and pre-supplementary ventrolateral prefrontal cortex for fearful bodies. although previous literature has mainly focused on the Besides the investigation of the effect of emotional congruency role of amygdala in face processing. 2005. the presence of fearful bodies can elicit the preparation of motor Specifically. These regions. expressions and involvement of the amygdala Thus. so an appropriate motor plan (Figure 3C and D) revealed a functional connectivity pattern be- can be finally prepared in the motor cortex (Devinsky et al. such as the encounter with a facial expressions. Therefore. the activity of SPL might be involved in providing a cor. 2007). 2007.. visual. This suggests that the processing of the face can stance.. 2004. 2018. congruent fear compound) the amygdala cial expression was the same in the compared compounds. regardless of the emotion conveyed by the face.. For has an essential role in orchestrating an appropriate response. together with the DLPFC. with the face. Goldberg et al. together with previous work. control and/or inhibition of was found for fearful compounds in the functional analysis. only the combin- cessed and a behavioural response is produced (Rudrauf et al. Rolls. 2017).. Peelen and Downing. Even when both faces were fearful. The activity of the ACC and OFC might also modulation in the processing of isolated faces in comparison to play a role in giving behavioural meaning to the perceived emo. de Borst information. entation of congruent fearful compounds. Vuilleumier and Beer et al. the motor plan to have an appropriate response to the given where the planning and execution of an adaptive action may be situation. involved in the signalling of other behaviourally relevant cues date the effect of emotional body postures on the processing of (Vuilleumier and Driver.oup. sequences (de Gelder et al. 142 | Social Cognitive and Affective Neuroscience. S1C). prefrontal and visual areas responded more strongly when faces are presented together with bodies rather than in isolation. Adding a fearful sustain an adaptive pattern of behaviour (de Gelder. this study aimed to eluci. This finding is in line with existing literature Conclusions stating that bodies not only provide emotional and social infor. 2014). Figure 3B). 1996.. sup- rect body representation. The functional activation analysis revealed fearful body (Peelen and Downing.. 2007). body while the other an incongruent happy body (CF > IH. Our functional activation analysis showed that motor.. that motor. the left amygdala presented more than fearful bodies (BH > BF. body to a fearful face increased the activity of motor and execu- Grèzes et al. be biased towards the expression conveyed in the bodies. posterior cingulate cortex (PCC).. 2005. fearful bodies caused more correlated activ- as shown by its increased functional connectivity with somato. pre- mation.. anterior lated happy bodies only activated visual areas significantly STG and visual areas. Moreover. inferior parietal lobule. stronger functional connectivity with preSMA and PCC. 2004. 1995. 2007). Van current body state with other sensory input. 2007. However. since no specific activations were found for fearful faces between the amygdala and these areas might be related to a when compared to happy faces (FH > FF. Vol. driving the activity activity of the preSMA and DLPFC (IF > FH. Fusar-Poli et al.. face–body compounds. since these areas are known to be literature supporting the key role of the amygdala in face pro- involved in emotional behaviour and also in the monitoring of cessing (Morris et al. tween the amygdala and other brain regions that could not 1995. Adolphs. 2017) in order to body compounds that presented fearful bodies. tive regions (CF > FF. superior parietal areas and the amygdala are essential in outside of the focus of attention and in emotional incongruence the processing of unambiguous face–body compounds. We found increased re- compounds. prefrontal. Supplementary Figure correct interpretation of the social and emotional cues con. the internal emotional state (Devinsky et al. 2005.e.. Figure 2F). 2007). 2014). the Driver. required (de Gelder et al. No. For in. 2005. compounds (Figure 3E–H). an initial evalu. as opposed to happy ones. The concurrent activation of prefrontal areas could and de Gelder.. Fearful tions that presented fearful bodies. presented together with bodies rather than in isolation. 1 arm-selective areas more strongly (Taylor et al. there is no rivalry between the emo. 2008. Haggard et al. den Stock and de Gelder. This tion of action cues may trigger regions of the motor cortex. 2016). More interestingly. no effect on motor structures also be necessary for the regulation.. but a clear effect was observed Of special interest were the results obtained for the condi- in the results of the PPI analysis (CH > CF. interesting findings resulted from the face– veyed by face–body compounds (Wang et al. Figure 2H). the comparison between compounds dorsal part of ACC and amygdala. 2004. Karnath et al.. In the case of a clear have been based solely on the emotion of the face. The observa. Vuilleumier et al. by integrating the information of the porting previous behavioural findings (Meeren et al. 2004... and representation of bodily states (insula) (Critchley. 2007). 2007). This finding is in line with previous tional face–body compounds. Likewise. had fearful faces but one of them presented a congruent fearful ation of their emotional congruency takes place (Meeren et al. 2005) and scenes (SPL) (Haggard et al. Grèzes et al.

Dolan. H. Freund. Stein. Tranel. stimulus basis for measuring perception of whole body ex- pression of emotions. A. Current Aviezer. (2005). (2000). (2016). A. J.. Snyder... (2000). H. M. 149–58.J... Orbitofrontal Neuroanatomy and Neurobiology to Clinical Neurology. Vroomen. 154–66. (1999). Biology. G. N. O. B. B. et al. 418. B. 34(6). The Journal of Comparative Transactions of the Royal Society B: Biological Sciences. Friston. Nature Reviews Neuroscience. (2006).. Nägele. D. D. B.J. Malach. In the case of an in. The perception of emotions by ear and by eye. M. Neuroimage. de Gelder. (2017). (2009).-O. de Borst. Preminger. Functional atlas of emotional faces processing: a Supplementary data are available at SCAN online. John. 289–311... Cortex.. Neural systems for recognizing emotion. The bodily expressive ac- the activity of motor areas even when the emotion in the face tion stimulus test (BEAST). A. Gerard. 284–92.D. 28–34. None declared. seem to play a role in the processing of unambiguous face–body de Gelder. H. Hadjikhani. Tye. 588–95. Funding Devinsky. The References emotion–action link? Naturalistic emotional stimuli preferen- Adolphs. ing bilateral damage to the human amygdala. (2015). R. Wiley Interdisciplinary Reviews: even when outside the focus of attention. B. Morgan. A. de Interpreting ambiguous social cues in unpredictable contexts.. M. 54. The Human Brain: Surface. G. Brain.. our results revealed that emotional de Gelder. Social Cognitive amygdala. ance imaging studies. R. Watson. the interaction of the amygdala with prefrontal. Poyo Solanas et al. The role of human basolateral amygdala in congruent face–body compound. Adolphs. Perceiving fear in dy- Impaired recognition of emotion in facial expressions follow. B. 669. of America. S. D. tially activate the human dorsal visual stream.. Tamietto. Greve.. K. affect. de Gelder. Hoffman. 2. Construction and validation of a was happy. sessment of the emotional content and level of congruency. M. R. Neta. (2012). Kim. Landi.2016.H. S. Abbamonte. B. de Gelder.... 7(5). R. P. emotion-cognition interactions. Gelder. J. (1994). The perception of body postures influence the processing of facial expressions. 223–33... 6(3).. 14(3). For instance. perception of faces in natural contexts. Davis.. In add.. H. The distributed 18(6). J. Journal of Cognitive Neuroscience. J. and Affective Neuroscience. 279–306.. E. B. and de Gelder. the processing of the face. Journal of Neuroscience. Knight.D. (2011). D.. (2016). S. Proceedings of the National Academy of Sciences of the United States motor and visual areas might be important for the correct as. (2016). somatosensory. (2012).. (2003). Gelder. Neurology.V.. 25(31). Blood Supply. ENEURO. de Gelder. K. 6. human neural system for face perception. A. number 295673). Y....I. P. Stein. 181. (2002). Rolls. ral network functions for threat perception. (2006).com/scan/article-abstract/13/1/135/4575135 by guest on 08 January 2018 .. van Honk. Baier... 11(8). 218–29. (2014). D. Hortensius. monitoring interoceptive signals and emotional (2004)... Frontiers in Psychology.. 517(7534). Towards the neurobiology of emotional body tioning of one’s own limbs mediated by the insular cortex? The language.T.J. namic body expressions. Cognition and Emotion. R. P. D.. J. Frontiers in Integrative Specifically. (2005). (2014).. 103(1). J. M. Carletti. R. Jeannerod.J. Cognitive Science. Hadjikhani.. B. R. The role of the basolateral amygdala in the Critchley. 959–67. J. D.A. 20.. (1995). 118(1). Grèzes. B. M. and Social Psychology. van Honk.. Placentino. (2015).A. and cognitive integration. van Honk.oup. fearful bodies increased de Gelder. morphology resulting from long-term amygdala damage. J. prefrontal and premotor areas Neuroscience. E. ing: faces with bodies tell the whole story.. Oxford: cortex and social behavior: integrating self-monitoring and Oxford University Press. and Programme for Research 2007–13 (ERC Grant agreement Three-Dimensional Sectional Anatomy. Goldberg.. an emo. for the preparation of behavioural action. de Borst.. Terburg. emotion in body expressions.. Neuroimage. H. O. From circuits to behaviour in the activity underlying affective body perception. 7134–8.W. Whalen. de Gelder. Seeing fearful body expres- 372(6507). Fink. Morgan. 493(1).. J. Philosophical ive... R. Downloaded from https://academic. J. Fear fosters flight: a mechanism for fear contagion processes to control for an adequate motor response. N. Stein. For instance. Rudrauf..S. Damasio. 16701–6.. (1997). D.J. sions activates the fusiform cortex and amygdala.M.. voxel-based meta-analysis of 105 functional magnetic reson- Conflict of interest. (2007). Wolpert. tional face combined with a congruent body enhances the activ. Damasio. B. 4(6). R. 11(5)... Furthermore.. Contributions of an- This work was supported by the European Research Council terior cingulate cortex to behaviour.. Scabini. 254–64. J.. Morgan. (ERC) under the European Union’s Seventh Framework Duvernoy. compounds.0314-16... D. ily expressions—a short review. Current Opinion in Neurobiology. M. R... body postures can also bias ambiguous social threat perception. B. B. M. Todorov. Mehta. 6(2).P. ition. Buechel. Vogt. de Social Cognitive and Affective Neuroscience. Gobbini.. T. Hortensius. Terburg. Leiguarda.. C. B.. Nature. Neural mechanisms of autonomic. Nature. Clear signals or mixed mes. 101(47)... (2012). D. B. The basolateral amygdalae and frontotempo- Social Cognitive and Affective Neuroscience. Pichon. B. Journal of Personality Haggard.J. Trope.C. 20150376. H. 13(24). 4(1). Psychophysiological and modulatory interactions in neuroimaging. Morrell. Hortensius. Attention and ity of cortical areas and the modulatory activity of the amygdala awareness each influence amygdala activity for dynamic bod- when compared to an incongruent face–body compound. temporal. 775–82. F. 242–9. sages: inter-individual emotion congruency modulates brain Janak... Neuroimage. New York: Springer- Verlag. 12(2).M. Hortensius. Moran. B. 35(2). Trends in Cognitive Boes. 169–77. when perceiving emotion expressed by a whole body. Changes in cortical Sciences. Terburg. Van den Stock. 871–9. D. Journal of Psychiatry and Neuroscience... 52. 84. P. 1299–309. Karnath. eNeuro. 2201–5. R. P. (2005). F. Supplementary data Fusar-Poli. Awareness of the func- de Gelder. 371(1693). M. Higher-Order Motor Disorders: From Beer. Holistic person process. A. 7(3). | 143 intentions. Haxby. Hallett. H. Morris.

anger.. (2007).A. Rapid inter. 150(3700). Van den Stock. logical interactions and functional connectivity..P. sions influence recognition of emotions in the face and voice. 2(5). (2012). Nature Neuroscience. pressions. P. David. (2004). No. (2007).T. Philosophical Transactions and Cognition. E. Büchel.M. E. Brain between human brain regions. Abdi. (2004). Vol.. lobe. M. E3305–14.L. J.. by attention and emotion: windows on causal interactions Rolls. Trends in Cognitive Sciences. F. J. 3(3–4). Goebel. J.. 13. J.. Sinke. Nugiel. J. Peelen. Zubin. Modulation of visual processing pressions. 604–9. M.E. (2008). The functions of the orbitofrontal cortex. Proceedings of the National Academy of Sciences of Translational Psychiatry. Social Van den Stock. M. Posamentier. Maintaining in- lysis of body and body part representations in the extrastriate ternal representations: the role of the human superior parietal and fusiform body areas. (2014)... et al. Pichon. 529–33. Richardson. Dolan. Distant influences of amygdala lesion on visual cor- Journal of Neuroscience. How affective information from faces and Peelen. A... cial expressions.X. NimStim set of facial expressions: judgments from untrained sponse in the human amygdala to fearful and happy facial ex. Social Van den Stock. body language. 1626–33.. Journal of Neurophysiology. de Gelder. e115.. 47–57.. P. S. R.. 2018. H.V. J. (2012). J. Downing. (1965).. C. (2009). S. Face identity matching is O’Reilly.W. J.. Olson. B.J. 837–55. M. Emotional modulation Vuilleumier.S.. 1271. J. Tools of the trade: psychophysio. B. 321–5. 362(1481). 7(5). Tanaka. N. 97(3).. Van den Stock. S. Neurobiology of Learning and Memory.. How brains beware: neural mechanisms of visual and motor areas by dynamic body expressions of of emotional attention. S. 114(16). P. 98(3). 2793–803. New York: Georg Thieme Verlag. D. (2003). P. et al.J. 7(3). 11–29..E. Lachaux. (2007). Gelder. (1996). 144 | Social Cognitive and Affective Neuroscience. (1998). of the Royal Society of London B: Biological Sciences. Vandenbulcke.J. I. influenced by emotions conveyed by face and body. Neuroscience.. Smith. 383(6603). Montoya. 9(12)..M. Nature Sutton. Metoki. A.R. P. J. J.. 1481–8. (1988).. 8(8).. research participants. Emotional information in Morris. Neuropsychology Review.. Dynamic neural architecture for social knowledge re- Human Brain: 3D Proportional System: An Approach to Cerebral trieval. R. A neuromodula. Grèzes. (1998). 16518–23. Atkinson. 28(11). The R. 8.C. The neural basis of visual scenes interacts in the brain. H.. (2012). Psychiatry Research. de Gelder. 55(1). 585–94. Nature Reviews Neuroscience. Body expres- Cognitive and Affective Neuroscience. C. (2005). Evoked-potential Neuroscience.. related structures rely on a two-pathway architecture.. Friston. Vuilleumier... Brain. Woolrich. United States of America.. Behrens. J. 13(3). 487. 113–43. Downing. de Cognitive and Affective Neuroscience. M. Rudrauf.. J. Collins. C. Proceedings of the National Academy of Sciences of the Imaging. B. et al. (2007). de Gelder. Nature. (2005). Leon. B. D. Husain. John. B. (2007).. 1187–8. A differential neural re..T. (2008). correlates of stimulus uncertainty.. Social Neuroscience. 102(45). 7(11). 9(10). Talairach.. et al. Processing faces and facial ex. Armony. Functional MRI ana. Emotional modulation of body-selective visual areas. J.. E. Wiggett. van Heijnsbergen.. Vuilleumier. Y. 31. Wang. 2(4). M. body and background hampers recognition memory for faces. C.. J. Taylor. 1 Meeren.. P. D. The Morris. Emotion. tical activation during emotional face processing. 168(3). by guest on 08 January 2018 .J.. J. P. 199–212. de Gelder. Frith.-P.C.. Goodbody. (2014).P.K. A. et al. D. J.. Wolpert. Co-Planar Stereotaxic Atlas of the (2017). A..oup. Tottenham. the United States of America. Hypervigilance perceptual integration of facial expression and emotional for fear after basolateral amygdala damage in humans. Driver. M. Perret. Vuilleumier. M.. B. 1(6). Tournoux. Koski. B. Braren. Frontiers in Johansen-Berg.. Rapid Terburg.W. H.. 636–48. actions between the ventral visual stream and emotion.E. 274–83. Downloaded from https://academic. tory role for the human amygdala in processing emotional fa. K. T. 53. J. Morgan.V. Andersson.. 121(1). 242–9.. Human Neuroscience. O.. Righart. de Gelder. Science. T.. Social Cognitive and Affective body perception...C.