Running Head: FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS !

Facial Emotion Recognition in Autism:

Details, Discrepancies and Future Directions

Jonathan Nickles

Northeastern University
FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS !2

Facial Emotion Recognition in Autism:

Details, Discrepancies and Future Directions

Autism spectrum disorder (ASD) is an exceedingly prevalent disease affecting over 3.5 million

people in the United States alone (Autism society, 2015). Because of this staggering number, nearly every

American knows, or has interacted with, someone who has been diagnosed. Understanding more about

how this disorder affects daily interactions is needed.

ASD is marked by abnormal social interaction, communication skills and repeated behaviors

(Grossman, Klin, Carter, & Volkmar, 2000). One of the primary results of ASD is the severe reduction in

drive to interact with those around them (Baron-Cohen, 1995). This reduction may be produced by an

impairment in the ability to perceive other’s emotional expressions, specifically emotion of the face

(Hobson 1993). Some emotion theories suggest that there are distinct facial expressions that map onto

basic emotions like happiness, sadness, fear, anger and disgust. Because nonverbal communication (e.g,

reading facial features) facilitates and maintains social interaction, a deficit in these skills can produce a

considerable setback in interaction (Feldman, 1982).

Facial emotion recognition (FER) has been a very popular area of study in the last three decades.

Despite the amount of research on this topic, the results are convoluted. Generally, research points to-

wards varying levels of impairment in FER for individuals with ASD depending on the specific task (e.g.

Baron-Cohen, 1991; Langdell, 1978; Teunisse & de Gelder, 1994). However, a number of studies have

failed to find any impairment of FER in those with ASD (e.g. Ogai et al., 2003; Ozonoff, Pennington, &

Rogers, 1990; Teunisse and Gelder, 1994). Despite these mixed results, the data that finds varying levels

of impairment are irrefutable.

Evidence for Emotion-Specific Impairments in ASD

All emotions are not equal when it comes to recognizing them through facial features. Emotions

can be broken down into internal and external emotions (Grossman, Klin, Carter, & Volkmar, 2000). In-

ternal emotions are those related to personal cognition which require mental schema to grasp. Internal

emotions include surprise and embarrassment. External emotions like happiness and sadness are seen in

more superficial situations that only require an understanding of contextual information. Simply put, the
FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS !3

difference between the two types of emotions are their underlying cause, external being the situation at

hand and internal being a belief about the world. Autistic subjects in studies of emotion cognition show a

marked inability to understand internal emotions (Baron-Cohen, 1991) which leads to impairment in fa-

cial recognition of the same emotions.

In recent studies researchers have looked at the ability of autistic subjects to recognize a number

of basic emotions. Their findings showed that there was notable impairment of all emotions, but an even

more severe deficit for negative emotions like fear, anger and disgust (Ashwin, Chapman, Colle, &

Baron-Cohen, 2007). The inability to recognize negative emotions is supported by the amygdala theory,

one of many theories that describes the cause, or partial cause of ASD. The theory links amygdala dys-

function to the impairment in recognition of negative emotions. Support for this theory comes from stud-

ies that that show an aversion to pursuing and maintaining direct eye contact (Grice et al., 2005), an abili-

ty of the amygdala. Furthermore, fMRI scans have shown decreased size as well as decreased activation

of the amygdala during different task demands (Baron-Cohen et al., 1999; Pierce, Müller, Ambrose,

Allen, & Courchesne, 2001). On a cellular level, autopsies performed on those with ASD have found

cell abnormalities within the amygdala (Bauman & Kemper, 1994). Lastly, in a study where the amygdala

of monkeys were lesioned, significant behavioral abnormalities mimicking the expressions of ASD were

found (Bachevalier, 1994). These studies support the amygdala theory, a likely cause as to why subjects

with ASD recognize negative emotions with significantly lower accuracy than positive emotions.

In authentic human interactions, emotions are not displayed to their fullest degree every single

time. Also, the majority of facial expressions that are displayed are not purely one emotion. For example

happiness and surprise can be seen when a deployed soldier returns home to his/her family. Anger and

disgust are often expressed in combination with one another when a partner is found to be unfaithful.

Consequently, researchers in 2010 combined differing intensities of emotion ranging from 20% to 100%

instead of only 100% like previous research. The task presented adolescents with ASD with a video of a

neutral expression morphing into an emotion with variable intensities. The researchers again found that

negative emotions were less accurately recognized than positive but more specifically, that anger, disgust,

and surprise were debilitated at different intensities. Specifically, surprise was only impaired at low levels
FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS !4

of intensity (20-40%), anger at low and medium (20-70%) and disgust at low medium and high

(20-100%) (6). The increased difficulty with disgust has been previously mentioned in studies (Ashwin,

Chapman, Colle, & Baron-Cohen, 2007) and may be a cause of insufficient eye gaze. As noted earlier,

individuals with ASD generally tend to avert their sight from others’ eyes. This can be especially prob-

lematic in emotion recognition because many of the clues the brain uses to identify emotion comes from

the eye region (Kussiko et al., 2009). Recognizing disgust, as opposed to the other basic emotions like

happiness, relies most heavily upon the eyes which is why there is impairment at all intensities.

In the same study, researchers combined the stimuli they presented by using a blended emotion

scale (BES). The results showed that ASD subjects interpreted ambiguous stimuli (an emotion and a neu-

tral expression blended) as negative as well as choosing negative emotions incorrectly (e.g. anger instead

of disgust) (Kuusikko et al., 2009). This may raise the question of why those on the spectrum are more

prone to attenuate towards negative emotions rather than positive, a bias which is also seen in other so-

cioemotional disorders including social anxiety and depression.

Happiness, on the other side of the spectrum, was found to be the emotion that was the easiest to

recognize. Control groups correctly picked it almost one hundred percent of the time and the ASD group

correctly chose it at above chance levels (Feldman, McGee, Mann, & Strain, 1993). This was the only

basic emotion in the test group to be chosen at above chance levels which is attributed to the fact that this

emotion is the first to be expressed, discriminated and recognized during development (Wellman, Harris,

Banerjee, & Sinclair. 2008). In a meta analysis of FER in ASD, the 95% confidence intervals for happi-

ness all spanned zero showing almost no difficulty across the studies. The only impairments that autistic

individuals faced were subtle differences in reaction time (Uljarevic & Hamilton, 2013) Nearly normal

levels of expression and recognition in happiness lends credit to the amygdala theory which affects only

negative emotions.

An explanation for the rate of recognition in specific emotions may be found in diagnosed indi-

viduals’ abnormal eye gaze. Not only do people with ASD avert their attention away from the eyes, but

they also attenuate to particular regions of the face instead of the face as a whole (Gross, 2004). Specifi-

cally, they are inclined to fixate their attention on the lower portion of the face more than the eyes and the

upper regions. While negative emotions rely upon the eyes and the detail of the eyebrows and forehead
FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS !5

for discrimination, positive emotions rely most heavily on the lower half of the face like the mouth and

cheeks. These gaze patterns would support the difficulty in recognition of negative emotion and apparent

ease in positive emotions.

To test these findings researchers presented stimuli in the form of partial face regions. A study

conducted in 1978, the very first study of FER in autism, took 10 autistic individuals and ten mentally

retarded non autistic individuals and looked at their ability to sort happy and sad faces. In the study, sub-

jects could discriminate both complete faces and just the lower half at similarly proficient levels. Howev-

er, when just the upper portion of the face was presented, there was a sharp decrease in recognition of the

test group (Langdell, 1978). These findings support the theory that autistic individuals avoid the upper

regions of the face, deteriorating the ability to recognize most emotions.

FER and ASD: Studies with Varying Stimuli

As researchers began to get a grasp of facial recognition and the basic emotions, they realized that

people with autism develop deep emotional schema that is specific to the human face (Gross, 2004). More

importantly was the fact that as the stimuli varied from the prototypical human face, the ability to recog-

nize emotion severely decreased. In fact, there was even a significant decrease between a random human

face and that of a family member, as well as peers versus a person outside of the subject’s age group

(Humphreys, Minshew, Leonard, & Behrmann, 2006; Langdell 1978). It must be noted that in these tasks

where the stimuli were manipulated, control groups also had a reduction in recognition ability but were

still able to identify emotions well above chance.

R.P. Hobson, a researcher well known for his work with autistic groups, conducted a number of

studies involving the recognition of emotion. One such study (1986), had test subjects sort a group of

photographs according to the emotion in the picture. Many of the faces in the picture had extraneous

stimuli in the form of different hats. Some hats were smaller and less intrusive while others were large

and overbearing. Not surprisingly, the participants paid little attention to the emotions on the faces and

began to categorize the photographs by the type of hat being worn (Hobson, 1986).

In another study conducted by Hobson and colleagues, the presentation of facial features was dis-

torted even further. Instead of the typical upright orientation, faces were presented upside down. Interest-

ingly enough, the test group was superior at identifying emotions on the upside down face compared to
FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS !6

the controls (Hobson, 1986). These findings may occur due to a permanence in the strategies used by con-

trol groups. During development, strategies for recognizing identity and emotion are refined; however,

when the face is rotated, the strategies no longer work. Autistic individuals lack complex strategies like

mental rotation and simply evaluate the face as-is. The reason why ASD subjects outperform the control

is because they have no regard for the meaning attached to the facial features. By sorting the meaningless

parts of the face they are just matching abstract patterns (Langdell, 1978). Although the same strategies

are used in both the upright and inverted tasks, the inverted faces are recognized at a lower rate because

they deviate from the prototypical face that autistic people score the highest on.

In the same study both groups were then presented with complete upright faces. As the study

continued, specific features of the face were blanked out. Starting with a blanked out mouth, test groups

maintained a high level of accuracy in recognition (this finding goes against Langell’s 1978 study that the

mouth is the critical feature for FER), but as subsequent features were removed the performance showed

further decrease (Hobson, 1986). The study ultimately reveals interesting advances on the strategies used

by those with ASD in FER. Specifically, the hierarchy of importance in features that are visibly available.

The final study that deviates from the typical human expression is one that looks at the difference

in recognition between human stimuli and nonhuman stimuli. In the study, 28 children with autism were

trained on a recognition task that presented the subjects with a picture of an emotional facial expression

(Gross, 2004). Subjects then had to match the expression with one of four options given (happy, sad, an-

gry, surprised. All also in photograph form). The stimuli in the study was either a human face, an orang-

utan face or a canine face. All the animal expressions were consistent with the description of primates and

canines as described in previous literature (e.g., Chevalier-Skolnikoff, 1973; Hall & Sharp, 1978). Find-

ings for this study paralleled many of the patterns already noted: children in the autistic group compared

to other control groups showed broad recognition deficiencies. Errors were higher when matching orang-

utan and canine faces especially from incorrectly choosing negative emotions when the correct answer

was either positive or neutral. Happiness was again the only emotion in which recognition rates were

above chance, this finding held true for canine faces. All other emotions, excluding sadness, were per-

ceived at a higher rate in human stimuli than animal; sadness was recognized at similarly low rates for

each species. The findings once again show that as the stimuli move further away from the typical human
FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS !7

expression, recognition of that emotion becomes worse. Although this holds true throughout many of the

studies, it is important to remember that there is a large amount of variability between each emotion.

Limitations and Future Directions

Despite over 100 published studies on emotion recognition in autism, the results remain mixed.

The findings of these studies vary for a number of reasons and through a recent meta analysis researchers

have uncovered a few of the larger dissimilarities between the studies,

First, participant sizes range from an N of 5 all the way to 97 (Uljarevic & Hamilton, 2013).

However only 15 of the 48 studies analyzed had groups with 20 participants or more. By conducting a

straightforward power analysis with a power of .95 and two independent groups (effect size =.08), the

studies require at least 35 participants in each group. Interestingly, the two studies with the largest sample

sizes both found no group differences. A trend in recent literature is toward larger sample sizes. Future

studies may clear some of the obscurity between studies.

A further problem is controlling for the actual participants and the tasks they do. The term ASD is

extremely broad and can be categorized into a number of different subgroups including: classical autism

(CA), high functioning (HF) autism and Asperger’s syndrome. Each of these categories has a different

level of mental functioning as well as FER ability. Many studies fail to categorize their subjects which can

lead to problems when comparing the data. A group of HF may show absolutely no deficit in FER while

on the same task someone who is a CA may show severe problems. The inverse dilemma may also occur

where the stimuli presented are either far too exaggerated (the ceiling effect) (Law Smith, Montagne, Per-

rett, Gill, & Gallagher, 2010), or far too subtle for any relevant effects to be shown (Humphreys, Min-

shew, Leonard, & Behrmann, 2006). One way a few studies have alleviated this problem was by control-

ling for IQ or verbal ability. By controlling for these details, the variability within the group decreases as

well as the mental ability between the test group and the control group. Future studies would benefit from

making note of the type of autistic participant they recruited as well as the range of IQ or verbal ability

which they controlled for.

Additionally, a wide variety of tasks have been implemented over the three decades this subject

has been studied. The two most popular are labelling and matching. In labelling participants view emo-
FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS !8

tional imagery and then either choose the correct answer from a pool of options or come up with their

own response. Matching asks the participant to choose the same emotion from a pool of options as the

emotion presented (Uljarevic & Hamilton, 2013). While the meta analysis did not find any significant

performance differences between the two tasks, there are a number of important differences that may play

smaller roles. Matching tasks may be able to be completed with some accuracy using contextual clues and

other compensatory strategies for recognition. Labeling tasks on the other hand present a number of op-

tions to choose from. Through simple elimination the correct answer may be guessed easily. Depending

on the individual participant, FER may vary between the two tasks.

In order for future studies to provide valid insight into this topic they must increase their sample

size and control for the participants they recruit (Uljarevic & Hamilton, 2013). If possible, it would be

beneficial for studies to include both matching and labelling tasks in order to account for individual dif-

ferences. Furthermore, a number of studies only used stimuli from a single person’s face. In order to de-

crease effects caused by familiarity and proximity to the prototype, it would be advisable to diversify and

regularize the stimuli used. Since the majority of the naturally occurring facial stimuli that is experienced

is moving, using dynamic stimuli in future studies may produce more applicable results. With dynamic

stimuli researchers would not only be able to look at the rate of recognition but the speed as well.

Studies of FER in the autistic community have resulted in wide range of results. The discrepan-

cies in these findings can be attributed to a variability in the studies (number of participants, type of task,

difficulty of the stimuli) as well as a variability in the disorder itself. In order to increase internal reliabili-

ty, researchers must increase their sample size and specify which type of autism they are studying.

Methodologically, studies should control for IQ and verbal ability, present a varying degree of difficulty

in the stimuli and include multiple tasks. Adherence to these suggestions in future studies will results in a

more clear picture of emotion recognition in autism.

FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS !9

References

Ashwin, C., Chapman, E., Colle, L., Baron-Cohen, S. (2007). Impaired recognition of negative

basic emotions in autism: A test of the amygdala theory. Social Neuroscience, 1(3-4),

349-363
FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS ! 10

Autism Society. (August, 2015). Facts and statistics. Retrieved from autism-society.org

Bachevalier, J. (1994). Medial temporal lobe structures in autism: A review of clinical and

experimental findings. Neuropsychologia, 32(6), 627-648

Bal, E., Harden, E., Lamb, D., Vaughan Van Hecke, A., Denver, J.W., Porges, S. W. (2010).

Emotion recognition in children with autism spectrum disorders: Relations to eye gaze

and autonomic state. Journal of Autism and Developmental Disorders, 40(3), 358-370

Baron-Cohen, Simon. (1991). Do people with autism understand what causes emotion? Child

Development, 62(2), 385-396

Baron-Cohen, S. (1995). Are children with autism blind to mentalistic significance of the eyes?

British Journal of Developmental Psychology, 13(4), 379-398

Baron-Cohen, S., Ring, H. A., Wheelwright, S., Bullmore, E. T., Brammer, M. J., Simmons, A.,

Williams, S. C. R. (1999). Social intelligence in the normal and autistic brain: An fMRI

study. European Journal of Neuroscience, 11(6), 1891-1898

Bauman, M., Kemper, T. (1994). The neurobiology of autism, Baltimore, Maryland: Johns

Hopkins University Press.

Bormann-Kischkel, C., Vilsmeier, M., Baude, B. (1995). The development of emotional con-

cepts in autism. Journal of Child Psychology and Psychiatry, 36(7), 1243-1259

Feldman, R. S. (1982). Development of nonverbal behavior in children. New York, New York:

Springer-Verlag Publishing

Feldman, R. S., McGee, G. G., Mann, L., Strain, P. (1993). Nonverbal affective decoding ability

in children with autism and in typical preschoolers. Journal of Early Intervention, 17(4),

341-350
FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS !11

Grice, S. J., Halit, H., Farroni, T., Baron-Cohen, S., Bolton, P., Johnson, M. H. (2005). Neural

correlates of eye-gaze detection in young children with autism. Cortex, 41(3), 342-353

Gross, T. F. (2004). The perception of four basic emotions in human and nonhuman faces by

children with autism and other developmental disabilities. Journal of Abnormal Child

Psychology, 32(5), 469-480

Grossman, J. B., Klin, A., Carter, A.S., Volkmar, F. R. (2000). Verbal bias in recognition of

facial emotions in children with asperger syndrome. Journal of Child Psychology and

Psychiatry, 41(3), 369-379

Hobson, R. P. (1986). The Autistic Child’s Appraisal Of Expressions Of Emotion. Journal of

Child Psychology and Psychiatry, 27: 321–342.

Hobson, P. R., Baron-Cohen, S., Flusberg, H. T., Cohen, D. J. (1993). Understanding persons:

The role of affect in understanding other minds: Perspectives from autism. Journal of

Child Psychology and Psychiatry, 12(2), 204-227

Humphreys, K., Minshew, N., Leonard, G. L., Behrmann, M. (2007). A fine-grained analysis of

facial expression processing in high-functioning adults with autism. Neuropsychologia,

45(4), 685-695

Kuusikko, S., Haapsamo, H., Verkasalo-Jansson, E., Hurtig, T., Mattila, M. L., Ebeling, H.,

...Moilanen, I. (2009). Emotion recognition in children and adolescents with autism

spectrum disorders. Journal of Autism and Developmental Disorders, 39(6), 938-945

Law Smith, M. J., Montagne, B., Perrett, D. I., Gill, M., Gallagher, L. (2010). Detecting subtle

facial emotion recognition deficits in high functioning autism using dynamic stimuli of

varying intensities. Neuropsychologia, 48(9), 2777-2781

FACIAL EMOTION RECOGNITION IN AUTISM: DETAILS, DIS ! 12

Langdell, T. (1978). Recognition of faces: An approach to the study of autism. The Journal of

Child Psychology and Psychiatry, 19(3), 255-268

Ogai, M., Matsumoto, H., Suzuki, K., Ozawa, F., Fukuda, R., Uchiyama, I...Takei, N. (2003).

fMRI study of recognition of facial expressions in high-functioning autistic patients.

Neuroreport, 14(4), 559-563

Ozonoff, S., Pennington, B. F., Rogers, S.J. (1990). Are there emotion perception deficits in

young autistic children? Journal of Child Psychology and Psychiatry, 31(5), 343-361

Pierce, K., Müller, R.A., Ambrose, J., Allen, G., Courchesne, E. (2001). Face processing occurs

outside the fusiform ‘face area’ in autism: Evidence from functional MRI. Brain: A

Journal of Neurology, 124(10), 2059-2073

Teunisse, J.P., de Gelder, B. (1994). Do autistic children have a generalized face processing

deficit? International Journal of Neuroscience, 77(3), 1-14

Uljarevic, M., Hamilton, A. (2013). Recognition of emotions in autism: A formal meta-analysis.

Journal of Autism and Developmental Disorders, 43(7), 1517-1526

Wellman, H. M., Harris, P. L., Banerjee, M., Sinclair, A. (2008). Early understanding of emotion:

Evidence from natural language. Cognition and Emotion, 9(2-3), 117-149