See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/343741952

AI Evaluation in Selection: Effects on Application and Pursuit Intentions

Article  in  Journal of Personnel Psychology · July 2020

DOI: 10.1027/1866-5888/a000258

CITATIONS READS

11 344

1 author:

Agata Mirowska
Rennes School of Business
11 PUBLICATIONS   28 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Prospective <2031>: Artificial Intelligence and Future of Work View project

Projet ECHOS View project

All content following this page was uploaded by Agata Mirowska on 14 March 2022.

The user has requested enhancement of the downloaded file.

 Research Note
https://econtent.hogrefe.com/doi/pdf/10.1027/1866-5888/a000258 - Agata Mirowska <agata.mirowska@rennes-sb.com> - Sunday, August 23, 2020 11:00:58 PM - IP Address:90.27.237.142

AI Evaluation in Selection
Effects on Application and Pursuit Intentions
Agata Mirowska

Department of Management & Organization, Rennes School of Business, Rennes, France

Abstract. This study investigates how information provided prior to the application stage of the selection process affects application intentions
toward the job and organization. Existing research has focused on applicants who have already entered into the selection process; however,
information revealed prior to application may cause candidates to self-select themselves out of the process. Utilizing a randomized exper-
imental design, participants read a job ad specifying that their prerecorded interviews would be reviewed by a human or an artificial intelligence-
based evaluator. The results show increased intentions to apply and pursue the job in the human evaluation condition.
Keywords: application intentions, intentions to pursue, technology in selection

During the recruitment process, potential applicants in-
terpret organizational signals (Bangerter, Roulin, & König,
2012) to make subjective perceptions of potential fit
(Chapman, Uggerslev, Carroll, Piasentin, & Jones, 2005;
Ehrhart & Ziegert, 2005). However, factors like the
adoption of new technologies may not be interpreted as
intended by all audiences (Drover, Wood, & Corbett, 2018),
increasing the risk of signal misinterpretation. This may
affect potential applicants’ choice to self-select into or
withdraw from the selection process (Born, Hiemstra, &
Oostrom, 2018; Langer, König, & Krause, 2017).
Interviews are one of the most widely used (Buehl &
Melchers, 2018; McCarthy et al., 2017), yet time and
resource intensive, selection tools. Not surprisingly,
technology designed to automate the interpretation of
verbal and nonverbal behaviors has attracted interest
(Chamorro-Premuzic, Akhtar, Winsborough, & Sherman,
2017). Most research to date on this subject has evaluated
applicant reactions to technology once they have entered
the selection process (e.g., Langer et al., 2017), ignoring the
potential effects on the makeup of the applicant pool. This
study applies signaling theory in a first step to under-
standing how awareness of the use of human versus ar-
tificial intelligence (AI)-based evaluation in the selection
process affects intentions to apply for and pursue a job with
a given organization. These dependent variables were
chosen as they represent preentry behavioral intentions,
which have been found to mediate the relationship be-
tween attitudinal variables and the choice of job or or-
ganization as an employer and may therefore impact the
quality of a company’s final hire (Chapman et al., 2005;
Gilliland, 1993; Highhouse, Lievens, & Sinar, 2003; Reeve
& Schultz, 2004). Throughout the paper, the term “AI-
based” will be used to refer to any technology that au-
tomates the decision stage, largely augmenting or re-
placing the human role in choosing among decision
options (Parasuraman, Sheridan, & Wickens, 2000).
Hypothesis Development
Signaling theory predicts that in situations of goal mis-
alignment and imperfect information, parties will seek to
learn about each other’s unseen characteristics by inter-
preting readily observable signals (Bangerter et al., 2012;
Spence, 1973). In the hiring context, the classic example is
education as a signal of applicant ability (Spence, 1973);
however, signals can include reputation, letters of rec-
ommendation, and even clothing – anything that is costly
or hard to fake for the sender (Bangerter et al., 2012). The
choice of selection tools may be used to signal what
employees can expect from a particular employer (Bowen
& Ostroff, 2004) and may be costly in terms of lost ap-
plications if not targeted properly.
However, such signals are only useful in as far as their
interpretation is in line with the intended message, which
may not always be the case (Bowen & Ostroff, 2004;
Drover et al., 2018). When signals are not yet well es-
tablished, the two sides must go through a process of
interpreting signals, validating this interpretation and
taking into account any adaptations from either side before
a steady state is reached where something can be safely
interpreted as a valid signal of a given characteristic
(Bangerter et al., 2012). With respect to the use of AI-based
evaluation technology, companies may initially aim to

Journal of Personnel Psychology (2020), 19(3), 142–149 © 2020 Hogrefe Publishing

https://doi.org/10.1027/1866-5888/a000258
 A. Mirowska, AI Evaluation in Selection
make the process more standardized, objective, and cost-
efficient or to signal financial fitness, efficiency, ob-
https://econtent.hogrefe.com/doi/pdf/10.1027/1866-5888/a000258 - Agata Mirowska <agata.mirowska@rennes-sb.com> - Sunday, August 23, 2020 11:00:58 PM - IP Address:90.27.237.142
jectivity, or a culture of innovation (Chamorro-Premuzic
et al., 2017). Alternatively, the adoption of this technology
may itself be an adaptation that increases cheating costs in
response to impression management and faking behaviors
(Bangerter et al., 2012; Buehl & Melchers, 2018).
Previous research on the use of digital and highly au-
tomated interviews shows that they are perceived as less
fair and controllable (Langer, König, & Fitili, 2019) and
may give rise to feelings of creepiness and greater privacy
concerns (Langer et al., 2017). Additionally, they may
hamper applicants’ ability to engage in various influence
tactics (Dürr & Klehe, 2018), as they do not provide
transparency and an unstructured format that would allow
for applicants to influence the progression of the interview
(Levashina & Campion, 2006). It is this lower explain-
ability of the decision systems underlying this technology
(Biran & Cotton, 2017), coupled with lower social band-
width and interactivity (Langer et al., 2019; Potosky,
2008), which may make applicants hesitant to subject
themselves to evaluation by AI-based technology.
Candidates may also extrapolate from the selection
process to company values and the expected work envi-
ronment (Bowen & Ostroff, 2004). Interviews are seen as
a communication exchange (Potosky, 2008), where both
verbal and nonverbal behaviors convey important infor-
mation (McCarthy et al., 2017). Human nonverbal be-
haviors are highly contextualized and, given that their
interpretation relies on understanding of context, hierar-
chy, and social structures (Bonaccio, O’Reilly, O’Sullivan,
& Chiocchio, 2016), are best evaluated by other human
beings. AI-based evaluation may be seen as ill-suited to
these aims and, absent other information about the
company, may be interpreted as signaling poor future
interpersonal treatment (Langer et al., 2017).
Hypothesis 1: Participants will report higher intention
to apply to jobs with human, rather than AI-based,
evaluators.
Hypothesis 2: Participants will report higher intentions
to pursue jobs with human, rather than AI-based,
evaluators.
Method
Participants
Participants were senior-level marketing students re-
cruited via a school-wide research participation platform
© 2020 Hogrefe Publishing
143
and given course credit in return for their participation.
This sample was representative of potential job seekers as
all students are required to complete end-of-year intern-
ships throughout the duration of their studies. Out of a
total of 259 participants who attempted the survey, 64
were not allowed to continue due to failing one of the
manipulation or attention checks, and another seven chose
not to complete the survey (i.e., stopped somewhere in the
process) and their responses were not included in the final
dataset. Finally, four students had taken the survey twice,
so their second responses were removed, resulting in a
final sample of 184 participants (111 female, Mage = 22.3,
SD = 1.54).
Procedure
Each participant was randomly assigned to read one of
two job ads, developed specifically for this study, fea-
turing a fictional company entitled “Misopton Tech-
nologies” to avoid confounding effects with familiarity
with existing organizations (Harrison & Stone, 2018). Job
ads were chosen as an example of the type of first contact
that an applicant may have with an organization. There is
some evidence that job advertisement content affects
perceptions of the organization; however, this effect was
found to be weaker for those with less work experience
(Walker, Feild, Giles, & Bernerth, 2008), which would
apply to the sample in this study. Additionally, there is
some evidence that female students were more likely to
apply for jobs based on the wording of the profile being
sought (Born & Taris, 2010); however, in the current
study, the job ads differed only regarding information
about the upcoming selection process: Participants were
told their recorded interviews would be reviewed either
by a company representative (Human Resources (HR)
evaluator condition; n = 97) or an AI interview assess-
ment software (AI evaluator condition; n = 87). Partici-
pants were not provided any additional information
about the selection process.
Measures
All measures were evaluated on a 1 (strongly disagree) to 5
(strongly agree) scale, unless noted otherwise.
Dependent Variables
Application intentions were captured using Taylor and
Bergmann’s (1987) two-item measure (α = .86). In-
tention to pursue a job with the organization was
measured using Highhouse et al.’s (2003) 5-item scale
(α = .77).
Journal of Personnel Psychology (2020), 19(3), 142–149
 144 A. Mirowska, AI Evaluation in Selection

Table 1. Descriptive statistics and correlations

Variable M SD 1 2 3 4 5 6 7 8
https://econtent.hogrefe.com/doi/pdf/10.1027/1866-5888/a000258 - Agata Mirowska <agata.mirowska@rennes-sb.com> - Sunday, August 23, 2020 11:00:58 PM - IP Address:90.27.237.142

1. Age 22.26 1.54 (–)

2. Gender 0.39 0.49 .05 (–)
3. Comfort with technology 4.19 0.76 .03 .02 (.89)
4. Organizational attractiveness 3.39 0.77 .00 .06 .03 (.83)
5. Organizational prestige 3.59 0.56 .03 .08 .06 .58** (.78)
6. HR vs. AI evaluator 0.47 0.50 .07 .02 .07 .02 .11 (–)
7. Application intentions 3.46 0.97 .06 .02 .01 .76** .48** .12 (.86)
8. Intentions to pursue 3.46 0.68 .02 .08 .01 .71** .61** .15* .69** (.77)
Note. N = 184. Internal consistency reliability coefficients (Cronbach’s α) appear in parentheses along the main diagonal. Gender: 0 = female, 1 = male; HR
evaluator = 0, AI evaluator = 1. AI = artificial intelligence. *p < .05, **p < .01.

Table 2. Summary of regression results

Dependent variable: application intentions Dependent variable: intentions to pursue
Variables B SE B β B SE B β B SE B β B SE B β
Control variables
Age .04 .03 .06 .04 .03 .06 .01 .02 .01 .00 .02 .01
Gender .06 .10 .03 .06 .10 .03 .03 .07 .02 .03 .07 .02
Comfort with technology .04 .06 .03 .05 .06 .04 .04 .04 .04 .05 .04 .06
Organizational attractiveness .91 .08 .72** .92 .08 .73** .47 .05 .54** .48 .05 .54**
Organizational prestige .12 .10 .07 .10 .10 .05 .36 .07 .30** .35 .07 .29**
Main effect
HR vs. AI evaluator .19 .09 .10† .14 .07 .11*
2
Adjusted R .57 .58 .55 .56
ΔR2 .01† .01*
ΔF 3.85 4.65
Cohen’s f2 .03 .03
Note. N = 184. Gender: 0 = female, 1 = male; HR evaluator = 0, AI evaluator = 1. AI = artificial intelligence. *p < .05, **p < .01, †p = .05.

Control Variables responses to these attention and manipulation check

Organizational attractiveness (α = .83) and prestige (α =
.78) were measured using Highhouse et al.’s (2003) 5-item
scales. These variables have been found to be important
predictors of recruitment outcomes and may serve as
antecedents of pursuit intentions (Chapman et al., 2005;
Highhouse et al., 2003; Uggerslev, Fassina, & Kraichy,
2012). Participants were also asked to report their age,
gender, and how comfortable they were with new tech-
nology [4-item scale adapted from Lassar, Mannolis, and
Lassar (2005), measured on a 1 (very uncomfortable) to 5
(very comfortable) scale; α = .89].
Analysis and Results
Several questions were included to ensure participants
had read the job ad and were aware of the selection
procedure to come; participants providing incorrect
questions were not included in the final sample. Table 1
shows descriptive statistics and correlations among study
variables, while Table 2 presents the results of the re-
gression analysis.
Considering standardized mean differences, the effect
of human versus AI-based evaluation on both application
intentions and intention to pursue was small but signifi-
cant, Cohen’s d = .24 and .30, respectively. When in-
cluding the control variables in the regression model, there
was a negative effect of AI evaluation on both dependent
variables. Being faced with an AI-based evaluation sig-
nificantly predicts lower application intentions (β = .10,
p = .05) and intentions to pursue the job (β = .11, p < .05).
Table 2 also reports the Cohen’s f2 for both models, as an
alternative method to quantify the effect size of the in-
dependent variable, taking into account the variation left
unexplained by the control variables. For both models,
Cohen’s f2 = .03, indicating a small but significant effect

Journal of Personnel Psychology (2020), 19(3), 142–149 © 2020 Hogrefe Publishing

 A. Mirowska, AI Evaluation in Selection
Table 3. Results of relative weight analysis
Dependent variable: application intentions
https://econtent.hogrefe.com/doi/pdf/10.1027/1866-5888/a000258 - Agata Mirowska <agata.mirowska@rennes-sb.com> - Sunday, August 23, 2020 11:00:58 PM - IP Address:90.27.237.142
Predictor variables RWraw (95% CI)
Age 0.00 ( 0.01, 0.03)
Gender 0.00 ( 0.01, 0.01)
Comfort with technology 0.00 ( 0.01, 0.02)
Organizational attractiveness 0.46 (0.38, 0.56)
Organizational prestige 0.15 (0.06, 0.18)
HR vs. AI evaluator 0.01 ( 0.00, 0.05)
Note. N = 184. AI = artificial intelligence; Rescaled RW = relative weight as a percentage of R2; RWraw = raw relative weight.
size (Selya, Rose, Dierker, Hedeker, & Mermelstein, 2012).
Interestingly, in both regressions, organizational attrac-
tiveness is a significant predictor of the dependent vari-
able, but organizational prestige is only significant when
intention to pursue is considered (β = .35, p < .01).
Although the high correlation between organizational
prestige and attractiveness gives rise to potential concerns
regarding multicollinearity, all variance inflation factor values
were below 1.55, well below the recommended conservative
threshold of 4 (Salmerón, J. Garcı́a, Garcı́a, & Martı́n, 2017).
Nevertheless, a relative weights analysis (Tonidandel &
LeBreton, 2015) was conducted to better interpret the con-
tribution of each variable in the model. As Table 3 shows, the
results largely mirror those reported above. The highest
weight in both models belongs to organizational attractive-
ness (78.07% and 60.37%), with organizational prestige also
showing a significant contribution, albeit this weight is
stronger in the model with the intention to pursue as the
dependent variable (36.18%) than with application intentions
(19.43%). Finally, the experimental condition shows a small
relative weight (1.73% and 2.66%), in both cases approaching
significance at the 5% level, although this technique has been
found to lead to higher than expected Type II error rates
(Tonidandel, LeBreton, & Johnson, 2009).
Due to the strong effects of organizational attraction and
prestige, and previous research (see, for example, Uggerslev
et al., 2012) supporting a potential mediating effect of the
attitudinal variables, organizational attraction, and prestige,
a post-hoc analysis was conducted. When looking at the me-
diating effect of organizational attraction, the direct effect
of the experimental condition was significant for both de-
pendent variables, application intentions (b = .20, SE = .09,
p < .05) and intentions to pursue (b = .18, SE = .07, p < .05).
However, the same cannot be said for the indirect effect as
the 95% CIs for both application intentions (indirect
effect = .03, 95% CI [ 0.24, 0.18]) and intentions to pur-
sue (indirect effect = .02, 95% CI [ 0.17, 0.12]) contained
zero. When considering organizational prestige as the medi-
ating variable, the direct effect of the experimental condition
© 2020 Hogrefe Publishing
145
Dependent variable: intentions to pursue
Rescaled RW (%) RWraw (95% CI) Rescaled RW (%)
0.54 0.00 ( 0.02, 0.01) 0.04
0.08 0.00 ( 0.01, 0.03) 0.46
0.14 0.00 ( 0.01, 0.03) 0.29
78.07 0.35 (0.28, 0.43) 60.37
19.43 0.21 (0.14, 0.29) 36.18
1.73 0.02 ( 0.00, 0.06) 2.66
100 100
was insignificant for application intentions (b = .13, SE = .13,
p > .05) and intention to pursue (b = .11, SE = .08, p > .05), as
were the indirect effects, application intentions (indirect
effect = .10, 95% CI [ 0.25, 0.03]), and intentions to pur-
sue (indirect effect = .09, 95% CI [ 0.22, 0.03]). Therefore,
there is no mediating effect of either attitudinal variable.
However, there appears to be a suppressor effect of organi-
zational attraction, as in both cases the direct effect of the
experimental condition is strengthened with the inclusion
of this attitudinal variable in the model.
Discussion
In general, the results here provide preliminary evidence
that information about the selection process may affect
individuals’ early decisions regarding applying for and
pursuing a job, precluding them from ever becoming a true
candidate. In both cases, these effects remained even
after controlling for the effects of organizational attrac-
tiveness and prestige, which prior meta-analyses have
established as important predictors of recruitment out-
comes (Chapman et al., 2005; Uggerslev et al., 2012). Not
surprisingly, organizational attractiveness and prestige
showed the greatest weight in explaining both dependent
variables; therefore, companies should continue to pay
attention to these attitudinal variables. However, a small
proportion of variance explained, as that shown for the
experimental condition, does not necessarily make this
variable unimportant (Tonidandel & LeBreton, 2015).
Interestingly, there was no direct effect of the experi-
mental condition on either of these attitudinal variables,
and the significant effect of the experimental condition
was only visible when attractiveness and prestige were
included in the model. It appears that their exclusion from
the model, particularly organizational attraction, hides the
true relationship between the experimental condition and
dependent variables (Cheung & Lau, 2008). This potential
Journal of Personnel Psychology (2020), 19(3), 142–149
 146
suppression effect implies that they are capturing some
variance of the dependent variables that is not related to
https://econtent.hogrefe.com/doi/pdf/10.1027/1866-5888/a000258 - Agata Mirowska <agata.mirowska@rennes-sb.com> - Sunday, August 23, 2020 11:00:58 PM - IP Address:90.27.237.142
the experimental condition, and their inclusion allows us
to clarify the relationship of interest (Watson, Clark,
Chmielewski, & Kotov, 2013). Therefore, it appears that
there is something about knowing one will be faced with
AI-aided versus human evaluation that is affecting be-
havioral intention, above and beyond the established path
via attitudes toward an organization, underscoring the
importance of considering additional factors beyond those
already established in the literature.
It appears that the selection process to come may act as
a signal, albeit one that is not interpreted the same way by
both companies and applicants. Applicants seemingly have
a negative view of AI-based evaluation technology, which
may affect their willingness to enter the applicant pool for
a given job or to fully pursue an application that is made. In
the case of application intentions, the effect was on the
cusp of significance, which may reflect an assumption that
there will be other opportunities within the selection
process for face-to-face interaction, thereby relieving
some initial applicant concerns. These are questions that
should be explored more deeply using qualitative research
designs to determine what the source of this aversion is
and what actions a company could take to overcome them.
However, the choice to actually work for an organization
may require the interpretation of signals to gain infor-
mation about the company’s values or culture as well as
how others will react to one’s association with the orga-
nization. The inclusion of an AI-based evaluation tool may
signal an impersonal nature, which turns applicants off
from pursuing employment with the organization. Orga-
nizational prestige captures perceived social consensus
about the value of a company’s characteristics (Highhouse
et al., 2003). Its significance only when predicting in-
tention to pursue, a more long-term oriented action in-
tention, supports the fact that one’s chosen workplace may
have repercussions on one’s social identity (Ashforth &
Mael, 1989; Carter & Highhouse, 2014).
Another potential explanation for the differing effects on
these two dependent variables may be found through the
lens of the theory of planned behavior (TPB). TPB argues
that intentions are affected not only by one’s attitudes and
the perceived social pressure to engage in a given behavior
but also by one’s perceived control over the success of that
behavior (Azjen, 1991). Being faced with an AI evaluation,
which may represent an unknown and seemingly less
controllable situation than a human evaluator, may affect
one’s perceived self-efficacy of achieving the ultimate goal
of working for the organization. The effect on application
intentions is smaller, as applying for the job is not affected
by the evaluation system to follow, although participants
may have still considered the utility of applying for a
Journal of Personnel Psychology (2020), 19(3), 142–149
A. Mirowska, AI Evaluation in Selection
selection process where they find it more difficult to gauge
their probability of success. Intentions to pursue the job,
however, require one to successfully pass through the
selection process; thereby, this intention may be affected
by one’s perceptions of lower control over the desired
outcome (being hired) due to an element outside of one’s
control (AI evaluation in the selection process).
Despite previous research showing that characteristics of
the recruitment and selection process affect attitudes toward
an organization (Uggerslev et al., 2012), there was no sig-
nificant effect of the knowledge of a human versus AI-based
evaluation on neither organizational attraction nor prestige
ratings. Therefore, it appears that people have a reaction to
the idea of AI evaluation independent of any ideas about the
specific organization utilizing this technology. This opens up
important avenues for future research, considering people’s
reactions to the idea of being evaluated by AI-based tech-
nologies in general, not just in recruitment contexts.
If this is the case, organizations should consider providing
more information about the selection process to temper the
signaling effects of only certain elements being known to
candidates. Such information may not only affect candi-
date reactions but also their subsequent performance and
motivation when moving through the selection process
(Truxillo, Bodner, Bertolino, Bauer, & Yonce, 2009). For
example, organizations could reveal various selection steps
to candidates early in the process, so candidates can clearly
see the proportion of technology-based versus interpersonal
assessments involved. Previous research shows that more
information can increase perceptions of openness and
fairness, increasing attraction to the company (Langer et al.,
2019; Truxillo et al., 2009). Additionally, as AI-based
evaluation is still a relatively new tool, companies may
provide further information regarding their choice to use
this technology in order to support the derivation and rit-
ualization processes of the signal, thereby increasing the
chances that this choice is interpreted correctly by potential
applicants. For example, organizations could consider ex-
plaining why they chose to use AI-based evaluation, clearly
tying it to the organizational attributes – efficiency, inno-
vation, objectivity – they believe it reflects.
Given that the organization presented was a fictional
one, with which the participants would not have had any
familiarity, it is possible that they did not have any existing
information structures, or “nodes,” in memory to which
to attach this information about the selection process
(Cable & Turban, 2001; Collins & Kanar, 2014). Their
ratings of organizational attraction and prestige may have
been global, preliminary attitudes albeit ones that did
not incorporate specific details that would be better as-
similated into the memory structures of participants fa-
miliar with the focal organization. This supports Cable and
Turban’s (2001) recommendations that companies should
© 2020 Hogrefe Publishing
 A. Mirowska, AI Evaluation in Selection
be aware of the level of organizational familiarity of their
company as an employer before deciding what additional
https://econtent.hogrefe.com/doi/pdf/10.1027/1866-5888/a000258 - Agata Mirowska <agata.mirowska@rennes-sb.com> - Sunday, August 23, 2020 11:00:58 PM - IP Address:90.27.237.142
information to provide. Comparative or longitudinal studies
should be conducted to further investigate whether such
documented effects as paying more attention to and re-
acting more strongly to familiar companies than unfamiliar
ones (Brooks, Highhouse, Russell, & Mohr, 2003; Dineen &
Allen, 2016), and better recall of information about familiar
versus unfamiliar companies at a later date (Cable &
Turban, 2003) also holds for information regarding the
selection process, particularly as it applies to the use of new
technologies. Potentially, the experimental condition would
have had a stronger effect on attitudes toward already
recognized employers. Alternatively, there may be some-
thing about the reactions to AI-assisted evaluation specif-
ically that operate independently of attitudes about the
company in question. For example, would a similar effect
hold for the use of AI evaluation in performance appraisals,
where individuals would presumably already be familiar
with the organization using the technology?
The self-report and cross-sectional nature of the data
give rise to concerns of common method variance, al-
though these are tempered somewhat by the finding of
significant differences between experimental conditions.
The study utilized a young sample and a technological
company, both of which may have positively affected the
acceptance of technological innovations in the selection
process (Fox & Connolly, 2018; Venkatesh & Bala, 2008).
If so, then these results may underreport true effects as
participants nonetheless showed a preference for human
evaluation. Finally, the study may have lacked realism due
to the fictional nature of the job ad, lack of a true appli-
cation and selection process, and inclusion of selection
process information in the ad itself. However, the study
design was meant to capture the phenomena of applicants
possessing information about the selection process, due to
either the popularity of employer review sites and social
media or compliance with ethical or legal requirements
(i.e., the General Data Protection Regulation).1 Given the
relatively small effect sizes seen here, potential moderators
such as industry or job level should be investigated. Ad-
ditionally, the suppression effect of the attitudinal vari-
ables was key in being able to see the effect of the
experimental condition. Although this could be a sign that
the true effect of AI evaluation on the intention variables is
obscured by the variance accounted for by the attitudinal
variables (Watson et al., 2013), further research may want
to better parcel out the various components of the atti-
tudinal and criterion variables to better understand which
1
I would like to thank an anonymous reviewer for this suggestion.
© 2020 Hogrefe Publishing
147
components are affected by elements of the selection
process, and in what way.
Future research should also consider mediators as po-
tential explanatory mechanisms of the results shown. The
current study was not well suited to capturing these medi-
ating mechanisms, but as scholars continue to investigate
reactions to various elements of the selection process, AI-
based technologies should be included in this list. Exploring
perceptions of justice and fairness is a particularly promis-
ing area, as these have been found to predict organiza-
tional attractiveness, job purist, and acceptance intentions
(McCarthy et al., 2017; Truxillo et al., 2009). Although AI-
based systems may provide the consistency of evaluation
that companies are looking for to maximize the validity of
their selection systems (Lievens & Sackett, 2017), predic-
tion and explainability are important features when con-
sidering the acceptance of AI technologies (Biran & Cotton,
2017). If potential candidates do not understand how they
are being evaluated in the selection system, they may not
consider the evaluation fair and be reluctant to enter the
selection process (Gilliland, 1993). This issue may remedy
itself as AI evaluation systems become more commonly
used, but organizations may want to consider providing
information about what they are looking to measure to
create more transparency around the process. Other ele-
ments of classic justice models, such as perceptions of in-
terpersonal justice via the lack of two-way communication
(Gilliland, 1993), also deserve attention.
A final point worth mentioning concerns the choice of
wording in the current study. Terms such as “AI-based
evaluation” may carry certain meaning for some indi-
viduals, ranging from extremely objective and reliable
measurement to an impersonal tool shrouded in secrecy.
However, this difference in meaning gets at the heart of
the signaling problem: Organizations may be drawn to
evaluations systems touted as AI-based, while potential
applicants may be turned off for the same reason. Future
research could investigate whether the terminology
around this new technology – alternatively calling it AI-
based or simply automated evaluation – leads to different
perceptions, reactions, and behavioral intentions.
References
Ashforth, B. E., & Mael, F. (1989). Social identity theory and the
organization. Academy of Management Review, 14, 20–39.
https://doi.org/10.5465/AMR.1989.4278999
Journal of Personnel Psychology (2020), 19(3), 142–149
 148
Azjen, I. (1991). The theory of planned behavior. Organizational
Behavior and Human Decision Processes, 50, 179–211. https://
https://econtent.hogrefe.com/doi/pdf/10.1027/1866-5888/a000258 - Agata Mirowska <agata.mirowska@rennes-sb.com> - Sunday, August 23, 2020 11:00:58 PM - IP Address:90.27.237.142
doi.org/10.1016/0749-5978(91)90020-T
Bangerter, A., Roulin, N., & König, C. J. (2012). Personnel selection
as a signaling game. Journal of Applied Psychology, 97, 719–738.
https://doi.org/10.1037/a0026078
Biran, O., & Cotton, C. (2017). Explanation and justification in ma-
chine learning: A survey. IJCAI-17 Workshop on Explainable AI
(XAI), 8, 8–13.
Bonaccio, S., O’Reilly, J., O’Sullivan, S. L., & Chiocchio, F. (2016).
Nonverbal behavior and communication in the workplace: A
review and an agenda for research. Journal of Management, 42,
1044–1074. https://doi.org/10.1177/0149206315621146
Born, M. Ph., Hiemstra, A. M. F., & Oostrom, J. K. (2018). Applicants’ role
as (pro-)active agents in the recruitment and selection process: A
frequently overlooked perspective. Journal of Personnel Psychol-
ogy, 17, 103–106. https://doi.org/10.1027/1866-5888/a000215
Born, M. Ph., & Taris, T. W. (2010). The impact of the wording of
employment advertisements on students’ inclination to apply
for a job. The Journal of Social Psychology, 150, 485–502.
https://doi.org/10.1080/00224540903365422
Bowen, D. E., & Ostroff, C. (2004). Understanding HRM-firm per-
formance linkages: The role of the “strength” of the HRM sys-
tem. The Academy of Management Review, 29, 203–221. https://
doi.org/10.2307/20159029
Brooks, M. E., Highhouse, S., Russell, S. S., & Mohr, D. C. (2003).
Familiarity, ambivalence, and firm reputation: Is corporate fame
a double-edged sword? Journal of Applied Psychology, 88,
904–914. https://doi.org/10.1037/0021-9010.88.5.904
Buehl, A.-K., & Melchers, K. G. (2018). Do attractiveness and
competition influence faking intentions in selection interviews?
Journal of Personnel Psychology, 17, 204–208. https://doi.org/
10.1027/1866-5888/a000208
Cable, D. M., & Turban, D. B. (2001). Establishing the dimensions,
sources and value of job seekers’ employer knowledge during
recruitment. In G. Ferris (Ed.), Research in personnel and human
resources management (Vol. 20, pp. 115–163). Bingley, UK: Em-
erald Group Publishing Limited.
Cable, D. M., & Turban, D. B. (2003). The value of organizational
reputation in the recruitment context: A brand-equity per-
spective. Journal of Applied Social Psychology, 33, 2244–2266.
https://doi.org/10.1111/j.1559-1816.2003.tb01883.x
Carter, N. T., & Highhouse, S. (2014). You will be known by the
company you keep: Understanding the social-identity concerns
of job seekers. In D. M. Cable & K. Y. T. Yu (Eds.), The Oxford
handbook of recruitment (pp. 454–462). New York, NY: Oxford
University Press.
Chamorro-Premuzic, T., Akhtar, R., Winsborough, D., & Sherman, R. A.
(2017). The datafication of talent: How technology is advancing the
science of human potential at work. Current Opinion in Behavioral
Sciences, 18, 13–16. https://doi.org/10.1016/j.cobeha.2017.04.007
Chapman, D. S., Uggerslev, K. L., Carroll, S. A., Piasentin, K. A., &
Jones, D. A. (2005). Applicant attraction to organizations and job
choice: A meta-analytic review of the correlates of recruiting
outcomes. Journal of Applied Psychology, 90, 928–944. https://
doi.org/10.1037/0021-9010.90.5.928
Cheung, G. W., & Lau, R. S. (2008). Testing mediation and sup-
pression effects of latent variables: Bootstrapping with struc-
tural equation models. Organizational Research Methods, 11,
296–325. https://doi.org/10.1177/1094428107300343
Collins, C. J., & Kanar, A. M. (2014). Employer brand equity and re-
cruitment research. In D. M. Cable & K. Y. T. Yu (Eds.), The Oxford
handbook of recruitment (pp. 284–297). New York, NY: Oxford
University Press.
Dineen, B. R., & Allen, D. G. (2016). Third party employment
branding: Human capital inflows and outflows following “Best
Journal of Personnel Psychology (2020), 19(3), 142–149
A. Mirowska, AI Evaluation in Selection
Places to Work” certifications. Academy of Management Jour-
nal, 59, 90–112. https://doi.org/10.5465/amj.2013.1091
Drover, W., Wood, M. S., & Corbett, A. C. (2018). Toward a cognitive
view of signalling theory: Individual attention and signal set
interpretation. Journal of Management Studies, 55, 209–231.
https://doi.org/10.1111/joms.12282
Dürr, D., & Klehe, U.-C. (2018). Using the theory of planned behavior
to predict faking in selection exercises varying in fidelity. Journal
of Personnel Psychology, 17, 155–160. https://doi.org/10.1027/
1866-5888/a000211
Ehrhart, K. H., & Ziegert, J. C. (2005). Why are individuals attracted
to organizations? Journal of Management, 31, 901–919. https://
doi.org/10.1177/0149206305279759
Fox, G., & Connolly, R. (2018). Mobile health technology adoption
across generations: Narrowing the digital divide. Information
Systems Journal, 28, 995–1019. https://doi.org/10.1111/isj.
12179
Gilliland, S. W. (1993). The perceived fairness of selection sys-
tems: An organizational justice perspective. Academy of
Management Review, 18, 694–734. https://doi.org/10.5465/
amr.1993.9402210155
Harrison, T., & Stone, D. L. (2018). Effects of organizational values
and employee contact on e-recruiting. Journal of Managerial
Psychology, 33, 311–324. https://doi.org/10.1108/JMP-03-2017-
0118
Highhouse, S., Lievens, F., & Sinar, E. F. (2003). Measuring attraction
to organizations. Educational and Psychological Measurement,
63, 986–1001. https://doi.org/10.1177/0013164403258403
Langer, M., König, C. J., & Fitili, A. (2019). Information as a double-
edged sword: The role of computer experience and information
on applicant reactions towards novel technologies for personnel
selection. Computers in Human Behavior, 81, 19–30. https://doi.
org/10.1016/j.chb.2017.11.036
Langer, M., König, C. J., & Krause, K. (2017). Examining digital in-
terviews for personnel selection: Applicant reactions and in-
terviewer ratings. International Journal of Selection and
Assessment, 25, 371–382. https://doi.org/10.1111/ijsa.12191
Levashina, J., & Campion, M. A. (2006). A model of faking likelihood
in the employment interview. International Journal of Selection
and Assessment, 14, 299–316. https://doi.org/10.1111/j.1468-
2389.2006.00353.x
Lievens, F., & Sackett, P. R. (2017). The effects of predictor method
factors on selection outcomes: A modular approach to per-
sonnel selection procedures. Journal of Applied Psychology,
102(1), 43–66. https://doi.org/10.1037/apl0000160
McCarthy, J. M., Bauer, T. N., Truxillo, D. M., Anderson, N. R., Costa,
A. C., & Ahmed, S. M. (2017). Applicant perspectives during
selection: A review addressing “so what?,” “what’s new?,” and
“where to next?”. Journal of Management, 43, 1693–1725.
https://doi.org/10.1177/0149206316681846
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model
for types and levels of human interaction with automation. IEEE
Transactions on Systems, Man, and Cybernetics – Part A: Sys-
tems and Humans, 30, 286–297. https://doi.org/10.1109/3468.
844354
Potosky, D. (2008). A conceptual framework for the role of the
administration medium in the personnel assessment process.
Academy of Management Review, 33, 629–648. https://doi.org/
10.5465/amr.2008.32465704
Reeve, C. L., & Schultz, L. (2004). Job-seeker reactions to selection
process information in job ads. International Journal of Selection
and Assessment, 12, 343–355. https://doi.org/10.1111/j.0965-
075X.2004.00289.x
Salmerón, R., Garcı́a, J., Garcı́a, C. B., & Martı́n, M. M. L. (2017). A
note about the corrected VIF. Statistical Papers, 58, 929–945.
https://doi.org/10.1007/s00362-015-0732-9
© 2020 Hogrefe Publishing
 A. Mirowska, AI Evaluation in Selection 149

Selya, A. S., Rose, J. S., Dierker, L. C., Hedeker, D., & Mermelstein, R. Walker, H. J., Feild, H. S., Giles, W. F., & Bernerth, J. B. (2008). The
J. (2012). A practical guide to calculating Cohen’s f2, a measure interactive effects of job advertisement characteristics and
https://econtent.hogrefe.com/doi/pdf/10.1027/1866-5888/a000258 - Agata Mirowska <agata.mirowska@rennes-sb.com> - Sunday, August 23, 2020 11:00:58 PM - IP Address:90.27.237.142

of local effect size, from PROC MIXED. Frontiers in Psychology, 3, applicant experience on reactions to recruitment messages.
111. https://doi.org/10.3389/fpsyg.2012.00111 Journal of Occupational and Organizational Psychology, 81,
Spence, M. (1973). Job market signaling. The Quarterly Journal of 619–638. https://doi.org/10.1348/096317907X252487
Economics, 87, 355–374. https://doi.org/10.2307/1882010 Watson, D., Clark, L. A., Chmielewski, M., & Kotov, R. (2013). The
Tonidandel, S., LeBreton, J. M., & Johnson, J. W. (2009). Determining value of suppressor effects in explicating the construct validity
the statistical significance of relative weights. Psychological of symptom measures. Psychological Assessment, 25, 929–941.
Methods, 14, 387–399. https://doi.org/10.1037/a0017735 https://doi.org/10.1037/a0032781
Tonidandel, S., & LeBreton, J. M. (2015). RWA web: A free, com-
prehensive, web-based, and user-friendly tool for relative History
weight analyses. Journal of Business and Psychology, 30, Received September 10, 2019
207–216. https://doi.org/10.1007/s10869-014-9351-z Revision received April 27, 2020
Truxillo, D. M., Bodner, T. E., Bertolino, M., Bauer, T. N., & Yonce, C. Accepted April 27, 2020
A. (2009). Effects of explanations on applicant reactions: A Published online August 19, 2020
meta-analytic review. International Journal of Selection and
Assessment, 17, 346–361. https://doi.org/10.1111/j.1468-2389. ORCID
2009.00478.x Agata Mirowska
Uggerslev, K. L., Fassina, N. E., & Kraichy, D. (2012). Recruiting  https://orcid.org/0000-0002-5475-5826
through the stages: A meta-analytic test of predictors of ap-
plicant attraction at different stages of the recruiting process. Agata Mirowska
Personnel Psychology, 65, 597–660. https://doi.org/10.1111/j. Rennes School of Business
1744-6570.2012.01254.x 2 rue Robert d’Arbrissel
Venkatesh, V., & Bala, H. (2008). Technology acceptance model 3 35065 Rennes
and a research agenda on interventions. Decision Sciences, 39, France
273–315. https://doi.org/10.1111/j.1540-5915.2008.00192.x Agata.mirowska@rennes-sb.com

© 2020 Hogrefe Publishing Journal of Personnel Psychology (2020), 19(3), 142–149

View publication stats