Evaluating Candidates and

Identifying the Short List

Sherri Irvin
Presidential Research Professor
of Philosophy and WGS
 Screening: systematic with clear criteria
• Template for each applicant with clearly defined criteria
as specified in the job ad
• Requirements
• Other attributes that will count in favor of hiring
Obstacles to identifying excellent candidates
• Implicit bias: tendency to underrate the credentials of
women, candidates of color, people with disabilities, and
other members of underrepresented groups
• Rater drift: tendency for evaluators’ standards to shift
over time, so similar credentials are rated differently
• Overemphasis on “fit”: tendency to discount the
achievements of people whose methods, topics, or social
identities are marginalized in the field
• Matthew Effect: tendency of further advantages to be
heaped on those who have experienced early
advantages, thereby inflating their credentials
Recent research shows that race, gender &
related factors play a major role in hiring
 Experiment on race and hiring
Resumes with African-American names received callbacks 50% less
often than those with white names.

Bertrand, Marianne, and Sendhil Mullainathan. “Are Emily and Greg more
employable than Lakisha and Jamal? A field experiment on labor market
discrimination.” The American Economic Review 94, no. 4 (2004): 991-1013.
Experiment on motherhood/gender and hiring

Resumes of mothers received significantly lower scores for

competence, organizational commitment and lower salary and
hiring recommendations (1.8x less).

Resumes of non-mothers received 2.1x the callbacks of

mothers.

Correll, Shelley J., and Stephen Benard. “Getting a job: Is there a motherhood
penalty?” American Journal of Sociology 112, no. 5 (2007): 1297-1339.
Effect of screened auditions on success of
female musicians

Use of screens during auditions accounts for 1/3 of the

increase in the number of female musicians in orchestras.

Goldin, Claudia, and Cecilia Rouse. Orchestrating impartiality: The impact of

“blind” auditions on female musicians. No. w5903. National Bureau of Economic
Research, 1997.
Experiment on gender and hiring in academic psychology

Men received a positive hiring recommendation >70% of the time;

women received a positive hiring recommendation only 55% of the
time.
Gender of reviewer did not matter. Women and men
discriminated equally regarding gender of candidate.

Steinpreis, Rhea E., Katie A. Anders, and Dawn Ritzke. “The impact of gender on the
review of the curricula vitae of job applicants and tenure candidates: A national
empirical study.” Sex Roles 41, no. 7-8 (1999): 509-528.
 Experiment on hiring of student lab
managers in university science labs
The female student was rated as less competent and less hireable.
Faculty offered less mentoring and proposed a 14% lower salary.

Gender and age of reviewer did not matter. Women and men
discriminated equally regarding gender of candidate.

Moss-Racusin, Corinne A., John F. Dovidio, Victoria L. Brescoll, Mark J. Graham, and
Jo Handelsman. "Science faculty’s subtle gender biases favor male students."
Proceedings of the National Academy of Sciences 109, no. 41 (2012): 16474-16479.
Experiment on race and gender in finalist
pools for academic jobs
“When there was only one woman or minority candidate in a pool of
four finalists, their odds of being hired were statistically zero.”

The odds increased dramatically with two women or two minority

candidates.

Johnson, Stefanie K., David R. Hekman, and Elsa T. Chan. “If there’s only one woman
in your candidate pool, there’s statistically no chance she’ll be hired.” Harvard
Business Review April 26, 2016.
 Implicit bias is exacerbated by
• Evaluator factors:
• cognitive load
• stress
• hurry
• fatigue, hunger, thirst
• belief that one is unbiased (probably false)
• Evaluation task factors:
• vague criteria
• lack of structure
• lack of accountability
Strategies
 Strategies: evaluator conditions
• Improve evaluator conditions
• Lighten workload
• Provide drinks and snacks
• Make sure dossiers are easy to access and review
 Strategies: counteracting bias
• Have a member of the diversity committee (DC) available
as a consultant on the search
• DC member can flag dossiers with potential bias triggers
for careful attention
• Social identity (gender, race, LGBTQ identity,
disability) where disclosed in the dossier
• Marginalized methods or topics
• Rationale: implicit bias sometimes functions by leading
us not to notice achievement
 Strategies: counteracting bias
• Consider anonymizing materials where feasible
• This must be done carefully (pronouns in letters, etc.)
• Can ask candidates to anonymize some materials
• DC member can assist or verify
• Not a cure-all: nature of research or organization
membership sometimes suggests social identity
• Be aware that submitted materials may be affected by
bias
• Letters of recommendation
• Student evaluations of teaching
 Strategies: accountability
• Create clear, objective criteria and well structured ratings
templates
• Points system where possible (if you deviate, be clear
about why)
• Justify short list by appeal to criteria

• This strategy helps with implicit bias, rater drift &

overemphasis on “fit”
Getting to the final list
 Initial interviews
• Consider forgoing initial (long-list) interviews
• Interview situation is not representative of actual job
tasks
• Submitted materials are most reliable source of info
• Vividness of interview swamps more reliable info
• Time and energy is better spent looking more
carefully at candidate materials
 Initial interviews
• If you decide to interview:
• Provide questions in advance
• Ask same questions of all candidates
• Provide consistent interviewing conditions
• Interviewer behavior affects performance
• Rank-order candidates prior to interview
• If you change your ranking, be clear about why
• What new info did you get that is relevant to
criteria?
 Final thoughts about the Matthew Effect
• Access to opportunities is facilitated by privilege (race,
gender, economic class, etc.)
• These opportunities tend to snowball
• careful mentorship in undergrad  top-ranked
graduate school  invitation to co-author  cushy
post-doc  three publications
 Experiment on which student queries
receive responses
Queries from women and students of color are far more likely to be
ignored, and requests for meetings more likely denied. The effect is
worse in disciplines where pay is higher.

Discrimination was just as bad in fields with higher ratios of

women and faculty of color.

Milkman, Katherine L., Modupe Akinola, and Dolly Chugh. "What happens before? A
field experiment exploring how pay and representation differentially shape bias on the
pathway into organizations." Journal of Applied Psychology 100, no. 6 (2015): 1678.
 Final thoughts about the Matthew Effect
• As you design criteria, look for ways of assessing merit
that do not simply codify the results of privilege
• One publication that challenges traditional
assumptions, published during a 4/4 teaching load, v.
three conventional publications published during a
teaching-free post-doc…