PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 48th ANNUAL MEETING—2004 2163

TYPE OF AUTOMATION FAILURE: THE EFFECTS ON TRUST AND

RELIANCE IN AUTOMATION

Jason D. Johnson, Julian Sanchez, Arthur D. Fisk, and Wendy A. Rogers

Georgia Institute of Technology
Atlanta, Georgia

Past automation research has focused primarily on machine-related factors (e.g.,

automation reliability) and human-related factors (e.g., accountability). Other
machine-related factors such as type of automation errors, misses or false alarms,
have been noticeably overlooked. These two automation errors correspond to
potential operator errors, omission (misses) and commission (false alarms), which
have proven to directly affect operators’ trust in automation. This proposed
research will examine how automation-error-type affects operator trust and begin
to develop baseline trust measures as they relate to error type and participant age.
It is expected that participants presented with more automation false alarms than
misses will experience a larger degradation of subjective trust than those
presented with equal numbers of false alarms and misses or more automation
misses than false alarms.

INTRODUCTION and Moray’s (1992) automated pasteurization plant

As the human race continues to progress, so
does the sophistication of the systems we use. From
military command and control systems and aircraft
to nuclear power plants and automobile assembly
lines, operators encounter automated systems on a
daily basis. Purely mechanical systems have been
mostly replaced by computers and circuit boards
allowing automated systems to perform tasks at
which human beings are historically poor
performers, such as monitoring during unengaging
tasks (Parasuraman, Mouloua, Molloy, & Hilburn,
1996). Although these systems are designed to
increase performance and decrease errors, they are
not perfect. In addition, “When humans are
involved, errors will be made, regardless of the
level of training, experience, or skill” (Park, 1997,
p. 151).
Understanding constructs such as trust, in
particular trust in automation, is imperative in
optimizing the overall relationship between the
operator and the system. Because of the potential
for devastating accidents to occur when trust in
automation is not allocated in appropriate levels
(i.e., over trusting automation despite other external
cues of a potential malfunction), much research has
been performed in the area of trust in automation
concerning function allocation. For instance, Lee
Downloaded from pro.sagepub.com at Freie Universitaet Berlin on May 2, 2015
experiments showed that when overall system
performance is low, based on efficiency and the
occurrence of faults, the operators’ trust in
automation was low (i.e., they generally opted for
manual control). Very little, however, has been
done to investigate how, if at all, the type of
automation error, false alarm or miss, affects an
operator’s trust in automation.
The decision to trust or not trust automation is
likely to be related to the more general construct of
trust. In the social psychology literature, trust is
traditionally defined on an interpersonal-level (e.g.,
“I do or do not trust a person or group”). In that
regard, each person must decide whether to 1) give
trust to another, and 2) act in a trust worthy manner
as to gain others’ trust. The first scenario is easily
extrapolated to relationships between operators and
machines. To illustrate, research on trust in teams
(Harris & Provis, 2000) categorized trust into two
levels: competence and intentions. The competence
component of trust referred to a person’s confidence
that someone else (or some group) has the ability to
perform in the manner in which they are expected to
or have advertised. The competence component is
easily mapped to an automated system. Operator’s
have both preconceived and developed over time
notions of an automated systems’ ability to perform,
or its reliability.
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 48th ANNUAL MEETING—2004
Trust can be described as a subjective measure
of one’s confidence in something or someone else.
In the context of automation, research has begun to
distinguish this subjective rating from its closely
related counterpart reliance. Trust refers to the
subjective reports of the operators or their feeling
about automation; whereas reliance is the objective
performance measures such as automation
utilization or task efficiency (Wiegmann, Rich, &
Zhang, 2001). Wiegmann et al. found that
participants’ subjective ratings of trust in
automation may be lower than their usage of the
automation; that is, trust ratings were less than 100
percent despite utilization rates of 100 percent.
Wiegmann et al. also showed, in general, that as the
reliability of the automated system decreased the
participants’ subjective ratings of trust in the system
decreased. The converse also appeared to apply.
Although it is important to understand the effects of
reliability in general, it is critical to understand how
other changes in an automated system, such as type
of automation error, affect an operator’s subjective
rating of trust.
Types of Errors
When an automated engine status indicator
malfunctions, one of two types of errors will occur.
First, if a system malfunctions and the automation
does not indicate a malfunction (the “signal” is not
detected), a miss has occurred. Second, if the
automation erroneously indicates a malfunction
when the system is working properly (a non-
existent signal is detected), a false alarm has
occurred. Research to date has not examined how
changes in these types of errors affect trust and
reliance in the automation.
Based upon the two types of automation errors,
operators can make two potential errors: omission
and commission errors. Omission errors occur when
an operator fails to respond to a system malfunction
in instances when the automation monitoring the
system fails to register or detect a malfunction.
Commission errors occur when an operator
inappropriately complies with the automation’s
directions concerning a system malfunction when
no real malfunction exists (Skitka, Mosier, &
Burdick, 2000).
Downloaded from pro.sagepub.com at Freie Universitaet Berlin on May 2, 2015
2164
Aging and automation
Along with the growing amount of research
examining trust and reliance in automation (e.g.,
Dzindolet, Peterson, Pomranky, Pierce, & Beck,
2003; Sanchez, Fisk, & Rogers, 2004; Wiegmann,
et al., 2001), more researchers are interested in how
these issues specifically relate to the effect of
operator age. As “baby boomers” approach old age,
the sheer number of older adults relying on
automation seems to be increasing nearly
exponentially. By determining age-related
differences in one’s approach to automation or
formulation of trust in automation, systems can be
designed and training programs developed with
those differences in mind. Some research suggests
that changes in trust developed based on automation
reliability differ based on age (e.g., Sanchez et al.,
2004). In their study, older adults showed a
significant loss of trust in automation when
reliability degraded from 100 to 80 percent and then
again from 80 to 60 percent. Younger adults on the
other hand only showed a statistically significant
loss in trust when reliability dropped from 100 to 80
percent. Understanding such age-related
differences in trust and reliance in automation are
important but underdeveloped. This present study
will add to our understanding of aging and factors
affecting use of automation.
PROPOSED METHOD
Participants
A total of 60 older and younger adults will
participate in this study. The 30 younger adults will
consist of males and females between the ages of 18
and 28, inclusive, from within the Georgia Institute
of Technology’s undergraduate psychology student
population. The 30 older adults will be male and
female volunteers between 65 and 75 years of age,
inclusive, from within Atlanta, GA and the
surrounding areas.
Proposed Design
Participants will be trained to perform a dual
task scenario in which they monitor a flight
simulator cockpit and respond to engine
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 48th ANNUAL MEETING—2004
malfunctions (the “engine task”) and monitor a
radar scope and report the number and type of
objects they see on the scope (the “radar task”). An
engine status indicator will be available to assist in
monitoring engine performance (see figure 1).
Participants will be instructed prior to training that
the engine status indicator might not be 100 percent
reliable and that it can be verified by selecting the
“view gauges” button which will display 100
percent reliable gauges.
Figure 1: Screen capture of simulator interface.
The type of automation failure, false alarm or
miss, will be a between-participants variable. Each
age subgroup, younger adults and older adults, will
be separated into three groups. Each group will
receive one of three experimental conditions:
predominantly misses, predominantly false alarms,
or equal number of misses and false alarms. The
dependent variables will be engine task and radar
task performance; the number of times the view
gauges button is pushed (objective trust measure);
subjective trust ratings; and the conditional
probabilities of operator error given an automation
false alarm, an automation miss, or an automation
error (either false alarm or miss).
Downloaded from pro.sagepub.com at Freie Universitaet Berlin on May 2, 2015
2165
Apparatus
The experiment will be conducted using a low-
fidelity cockpit simulator. Participants will interact
with the simulation using a standard two-button
mouse. Participants will be required to monitor
engine performance and reset engines when they
malfunction. In conjunction with monitoring
engine performance in the top half of the screen,
each participant will also monitor a radar scope on
the bottom half of the screen and click on symbols
as they appear on the radar scope.
Engine task. The engine task consists of monitoring
the engines to ascertain when an engine malfunction
has occurred. The engine status indicator is
designed to aid the operators in determining if an
engine is malfunctioning. It will have a reliability
of 80 percent (i.e., the engine status indicator will
indicate a true engine malfunction 80 percent of the
time). The remaining 20 percent will consist of
false alarms and misses in proportions that are
condition specific: 75/25, 25/75, and 50/50 false
alarm to miss respectively. The 80 percent
reliability of the engine status indicator was chosen
to replicate a specific condition in previous research
conducted by Sanchez et al. (2004).
Radar Task. The radar task will consist of a radar
scope that will display four different symbols. The
participants will be instructed to select symbols
each time they see them on the radar scope. Their
goal will be to depress a symbol’s corresponding
button when they first notice a symbol has appeared
on the radar screen (See figure 1). The performance
data collected from the radar task will include the
ratio of symbols selected to symbols displayed.
Questionnaire
The experiment will include a basic demographic
questionnaire and subjective trust questionnaires.
The trust questionnaires will be administered after
training, after block one, and after block two. The
trust questionnaire will provide information about
each participant’s overall trust in automated
systems, trust in this particular automated system,
and changes in trust based on the type and
proportion of automation errors they encounter.
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 48th ANNUAL MEETING—2004
Procedure
Upon beginning the study and providing
informed consent, each participant will complete a
demographic questionnaire and an initial trust
questionnaire. Participants will perform a seven
minute training block to learn the engine and the
radar tasks. Following the completion of training,
participants will perform two experimental blocks,
lasting 20 minutes each. At the completion of each
block, participants will be given the trust
questionnaire and performance feedback. Each
participant will begin each block of testing with
1000 points. They will lose five points for each
“mistake” (e.g., missing a symbol on the radar task
or failing to reset a malfunction on the engine task)
and 1 point each time they press the view gauges
button. Participants will be provided with a
performance feedback scale to internally rank their
performance consisting of point ranges associated
with popular “fighter jock” labels. Scores between
1,000 and 950 will be classified as “Top Gun”, 949
to 900 as “Ace”, 899 to 850 as “Instructor Pilot”,
849 to 800 as “Pilot Trainee”, and less than 800 as
“Cadet”.
EXPECTED RESULTS
Analyses of variance (ANOVAs) will be performed
to assess the main effects of type of automation
error and age and any interaction effects. In
addition, conditional probabilities will be calculated
to determine 1) the probability of an operator error
given the occurrence of an automation error, 2) the
probability of an operator error given the
occurrence of a false alarm, and 3) the probability
of an operator error given the occurrence of a miss.
It is expected that the majority automation
false alarms condition will produce larger drops in
subjective trust than will the majority automation
misses condition. The following passage from
Shlomo Breznitz’s Cry Wolf: The Psychology of
False Alarms (1984) highlights the various potential
consequences of continued false alarms on an
operator.
Each false alarm reduces the credibility of a
warning system. The credibility loss
following a false alarm episode has serious
ramifications to behavior in a variety of
Downloaded from pro.sagepub.com at Freie Universitaet Berlin on May 2, 2015
2166
response channels. Thus, future similar
alerts may receive less attention. They may
elicit weaker fear reactions. The threat may
be perceived as less intense or less
probable. People may overestimate their
ability to cope with the danger if and when it
materializes. Or, most important, they may
reduce their willingness to engage in
protective behavior. (p.11)
Other researchers in addition to Breznitz’s (e.g.,
Bliss & Dunn, 2000) have supported the notion that
persistent or pervasive false alarms negatively affect
operator trust in automated systems. Very little,
however, has been done to examine how, if at all,
misses affect trust. One might imagine that
automation misses would have a lesser effect on
operator trust because the operator is experiencing
less interaction with the system as a result of the
automation not indicating the system malfunctions.
DISCUSSION
Trust and reliance in automation are dynamic
traits that are significantly affected by many factors.
Much research has been conducted to date to
examine how some of these factors affect an
operator’s trust in increasingly automated systems,
such as self-confidence, reliability, or prior
knowledge of impending automation failure (e.g.,
Dzindolet et al, 2003; Moray, Inagaki, & Itoh, 2000;
Sanchez et al., 2004;Wiegmann et al., 2001). Very
little has been done to examine how the type of
automation failure affects an operator’s trust and
reliance in automation. This proposed study will
establish the foundation upon which more elaborate
studies can build. By holding the scenario
description and reliability constant (i.e., referring
only to malfunctions not specifically false alarms or
misses and maintaining an overall automation
reliability of 80 percent) this study will isolate the
type of automation failure and its effect on trust and
reliance in the decision aid. One potential finding is
that false alarms will produce lower levels of
subjective trust. Increasing the number of false
alarms committed by the decision aid will increase
the number of messages displayed by the decision
aid. After a false alarm or two, the participants may
be more likely to press the “engine gauges” button
to confirm each subsequence decision aid
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 48th ANNUAL MEETING—2004
malfunction indication. Misses, on the other hand,
are completely unknown until the decision aid
displays the message stating that the engine
malfunctioned and was reset (i.e., “out of time”
displayed in the action results window). The only
way for the participants to confirm a miss is to try
to completely prevent it (i.e., continuously press the
view gauges button in an attempt to catch the miss
when it is occurring). Lastly, this research may find
that trust does not vary with the type of error
experienced by the participants.
Proposed follow-up research will need to be
conducted to examine some of the questions this
study is not designed to answer. Future work
should investigate the effects of changing the
scenario in which the participants find themselves,
such as monitoring a nuclear reactor or conducting
DNA tests for a criminal trial. By doing this,
participants will be put in situations which appear to
foster tolerance of one type of error over the other.
Also of interest, and closely related to the scenario,
is the degree to which the cost of the automation
failure is varied. Lastly, a multifactor experiment
should be conducted in which all of the
aforementioned factors are systematically varied to
determine their levels of interaction. Those results
coupled with this single factor experiment will
provide a more complete picture of the complex
dynamics involved with determining and predicting
operator trust and reliance in automation.
ACKNOWLEDGMENTS
The research team would like to extend a very
special thanks to Neta Ezra for her contributions to
programming the interface. This research was
supported in part by contributions from Deere &
Company and we thank Jerry Duncan and Bruce
Newendorp for their support and advice on this
research. This research was also supported in part
by a grant from the National Institute of Health
(National Institute of Aging) Grant P01 AG17211
Downloaded from pro.sagepub.com at Freie Universitaet Berlin on May 2, 2015
2167
under the auspices of the Center for Research and
Education on Aging and Technology Enhancement
(CREATE).
REFERENCES
Bliss, J. P., & Dunn, M. C. (2000). Behavioral implications of
alarm mistrust as a function of task workload.
Ergonomic, 43, 1283-1300.
Breznitz, S. (1984). Cry wolf: the psychology of false alarms.
Lawrence Erlbaum Associates: Hillsdale, NJ.
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L.
G., & Beck, H. P. (2003). The role of trust in automation
reliance. International Journal of Human-Computer
Studies, 58, 697-718.
Harris, H., & Provis, C. (2000). Teams, trust and norms:
there’s more to teams as a manufacturing management
system than some might think. Proceedings of the 8th
International Conference on Manufacturing
Engineering. Sydney, Australia, 1-5. (Available from
http://www.smartlink.net.au/library/harris/teamstrustnor
ms.pdf).
Moray, N., Inagaki, T., & Itoh, M. (2000). Adaptive
automation, trust, and self-confidence in fault
management of time-critical tasks. Journal of
Experimental Psychology: Applied, 6(1), 44-58.
Parasuraman, R., Mouloua, M., Molloy, R., & Hilburn, B.
(1996). Monitoring of automated systems. In R.
Parasuraman & M. Mouloua (Eds.), Automation and
human performance: theory and applications (pp. 91-
115). Mahwah, NJ: Lawrence Erlbaum Associates.
Parks, K. S. (1997). Human error. In G. Salvendy (Ed.),
Handbook of human factors and ergonomics (pp. 150-
173). New York: Wiley-Interscience.
Sanchez, J., Fisk, A. D., & Rogers, W. A. (2004, March). Age-
related and reliability related effects on trust of a
decision support aid. Poster session presented at the
2004 Human Performance, Situation Awareness and
Automation Technology Conference, Daytona Beach,
FL.
Skitka, L., Mosier, K., & Burdick, M. (2000). Accountability
and automation bias. International Journal of Human-
Computer Studies, 52, 701-717.
Wiegmann, D., Rich, A., & Zhang, H. (2001). Automated
diagnostic aids: the effects of aid reliability on users’
trust and reliance. Theoretical Issues in Ergonomics
Science, 2(4), 352-367.