Gender Bias in the Reading the Mind in the Eyes Test

The Reading the Mind in the Eyes test (RME) was developed by Simon Baron-Cohen to test aspects of
his extreme male brain theory of autism. Recently, it has become popular with researchers studying
affective empathy and autism, and is now available online in various places, mostly as a diagnostic tool
for adults on the autism spectrum. It has been revised, and the current version contains 36 items, each
with four possible responses in a multiple choice format. Unfortunately, even in the revised version, the
test items are highly gender biased which could affect its accuracy even though it has decent test-retest
reliability. I have analyzed the content of the test items, and broken them down by gender in order to
highlight this problem and bring it to the attention of psychology researchers.

Restricted range of female images

The images are cropped black and white stills of actors in movies. These are not taken from real life,
and are not representative of the population although the items are balanced in terms of number of male
and female eyes (18 female and 18 male). There is a much greater range of ages represented by the
male faces as compared to the female faces. This may be due to the restricted pool (movies) that the
items were drawn from. All the female eyes are model-like in their symmetry, and all are made up with
eye makeup and have plucked eyebrows. In other words, all the female eyes are conforming to a
gender stereotype of feminine beauty as seen in movies.

1
Restricted range of female emotional expression

I made a list of all the response choices (77 different choices) and found that there were only 48
different choices presented to the test subject as possible responses for the female eyes while there were
54 different choices presented for the male eyes. Thus the range of possible emotions for the female
eyes was restricted as compared to the male eyes.

Range and Frequency of Response Choices

Female Male Female Male
accusing X friendly XXX
affectionate XX grateful XXX
aghast XXXXX X guilty XX
alarmed X XX horrified X X
amused XX XX hostile XX XX
annoyed X X impatient XX
anticipating X imploring X
anxious X incredulous X
apologetic X indecisive X
arrogant XX X indifferent X
ashamed X XX insisting XX XX
baffled X interested XX
bewildered X irritated XX XXXX
bored XX X joking XXXX X
cautious X X nervous X X
comforting X panicked XX
concerned X pensive X
confident X playful XX
confused X preoccupied XX
contemplative XX puzzled X
contented X X reassuring X
convinced X reflective X
curious X regretful X
decisive X X relaxed X
defiant X relieved X
depressed X sarcastic X X
desire X serious X
despondent X X shy XXX
disappointed X X skeptical X
dispirited X XX suspicious X
distrustful X sympathetic X
dominant X tentative X
doubtful X terrified X XX
embarrassed X XX thoughtful X
encouraging X X threatening X
excited XX uneasy X
fantasizing XX X upset X
flirtatious X X worried X
flustered XX
Total: 77 48 54

2
Similarly, among the correct responses, there were 18 different correct responses for the male eyes (a
unique response for each item), and only 15 for the female eyes since three of the female items
(“fantasizing”, “preoccupied”, and “interested”) were duplicated. Again, a greater emotional range
was available to describe the male eyes. There was apparently no effort made to ensure parity between
the range and types of emotions expressed in the male and female items.

Range of Correct Responses

Female Male
accusing X
anticipating X
cautious X X
concerned X
confident X
contemplative X
decisive X
defiant X
desire X
despondent X
distrustful X
doubtful X
fantasizing XX
flirtatious X
friendly X
hostile X
insisting X
interested XX
nervous X
pensive X
playful X
preoccupied XX
reflective X
regretful X
serious X
skeptical X
suspicious X
tentative X
thoughtful X
uneasy X
upset X
worried X

Total: 32 15 18

3
Sexualized content of female items

Among the correct responses for female eyes, we find: “desire”, “flirtatious”, “interested” (twice), and
“playful”. Similar emotions in the male eyes are described as “anticipating” and “friendly”. The
correct responses for the female eyes have a strong bias toward expressing sexual interest directed
toward the viewer. This perception is reinforced by the camera angles and lighting that were used in
the movies from which the items were drawn.

Interested Friendly

Also among the correct responses for the female eyes are: “contemplative”, “fantasizing” (twice),
“preoccupied” (twice), and “reflective”. The eyes in those items are not expressing much emotion.
Similar items for male eyes were described as “pensive” and “thoughtful”. All of these indicate
inwardly directed attention. Fantasizing is not really an emotional state; it is an activity during which
one could experience a variety of emotional states. When someone's attention is directed inward, less
communication about their emotional state is directed outward making the actual emotional states
depicted in these items ambiguous.

Fantasizing Thoughtful

Answering these ambiguous items correctly is more about eliminating answers than making a
distinction between “thoughtful” and “fantasizing”. Also, ambiguous items may muddy the results
since test subjects may be more likely to project their own emotional states or be more suggestible to
response items. Six female items (out of 18) were ambiguous as compared to two (out of 18) for the
male items, so fully 33% of the female items were ambiguous compared to 11% of the male items.
This cluster of similar items among the correct answers for the female eyes further restricts the range of
emotions expressed.

Using the RME to look for gender differences in empathy could be very problematic. One study
concludes that men have more trouble reading the emotions of women than of other men based on their
performance on the RME. Given that there is a significantly higher percentage of ambiguous items
among the female eyes, and the range of emotions expressed in the female items is limited, I'm not sure
that is a valid conclusion. If the RME is to be used for testing hypotheses about autism and gender or

4
gender and affective empathy, it needs to be redesigned with careful attention to gender parity in both
the range of different ages and facial types presented as well as the range and types of emotions
represented.

Sources

Baron-Cohen S. The extreme male brain theory of autism. Trends in Cognitive Science. 2002; 6(6):248-
254.

Baron-Cohen S, Wheelwright S, Hill J, Raste Y, Plumb I. “Reading the Mind in the Eyes” test revised
version: a study with normal adults, and adults with Asperger syndrome or high-functioning autism.
Journal of Child Psychology and Psychiatry. 2001; 42: 241-251.

Best C, Minshew N, Strauss M. Gender discrimination of eyes and mouths by individuals with autism.
Autism Research. 2010; 3(2): 88-93.

Fernández-Abascal E, Cabello R, Fernández-Berrocal P, Baron-Cohen S. Test-retest reliability of the

'Reading the Mind in the Eyes' test: a one-year follow-up study. Molecular Autism. 2013; 4:33.

Schiffer B, Pawliczek C, Müller B, Gizewski E, Walter H. Why Don't Men Understand Women?
Altered Neural Networks for Reading the Language of Male and Female Eyes. PLOS One. 8(4):
e60278.