Proceedings of the Human

TRUST MEASUREMENT in Factors and Ergonomics INTERACTION:

HUMAN-AUTOMATION Society 2019 Annual Meeting REVIEW, San Jose State University
A SYSTEMATIC Page 1 of 1595
5

Trust Measurement in Human–Automation Interaction: A Systematic Review

Matthew Brzowski, Dan Nathan-Roberts Ph.D.

Industrial and Systems Engineering, San José State University

This systematic review summarizes current measurements of trust in human-automation interaction. A total
of 217 articles were found, and it was determined that 44 articles contained relevant information and met
inclusion criteria. The results of the review showed that 75% (n = 33) of articles used subjective measures
of trust only, and 41% (n = 18) used researcher-defined methods of measuring trust instead of peer-
reviewed and validated scales. Of 10 defined industries, the highest number of articles (n = 14) were
assigned to the automotive industry, followed by aviation, military, and security (n = 6). The automated
systems studied in relevant articles were decision aids, automated control and navigation systems, and
process control systems. This review showed that research of trust in human-automation interaction (1) has
the tendency to use subjective measures of trust as the primary or only measure, (2) has the tendency to
individually define trust and how it is measured, and (3) is heavily composed of research on automotive
automation. Best practices and future research are discussed.

INTRODUCTION Without consideration for trust in these automated

Trust is difficult to define, yet the body of research on
human–automation interaction (HAI) dating back to 1973 and
before suggests that trust could play a large role in the
successful adoption and subsequent use of automated systems
(Lee & Moray, 1994). The current literature on HAI lacks a
consensus on a formal definition of trust, and often,
investigators choose to define it individually.
The difficulty of measuring trust is somewhat due to its
multifaceted nature. Larzalere and Huston (1980) observed
that factors such as predictability, reliability, and
dependability play important parts in trust, but other studies
have shown that elements such as accurate mental models of
Copyright 2019 by Human Factors and Ergonomics Society. DOI 10.1177/1071181319631462
systems, America’s industries will find that automation
adoption is difficult for employees, and high-risk industries
such as the military and medicine will find users neglecting to
use such systems in favor of their own judgement, even when
it may be at a fundamental disadvantage (Alexander et al.,
1994; Crowley et al., 2013).
The current body of literature on human–automation
interaction fails to consistently define trust and to measure it.
To validate measures of trust in human–system interaction,
researchers must first standardize definitions of trust and the
methods to measure it. This review aims to provide a
foundation for trust measurement in HAI in the hopes of
stating its importance and reiterating the need to standardize

the system (Muir, 1987) and the system’s performance (Lee,
1991) could have an equal role in determining trust (Balfe et
al., 2018).
A 2013 paper published in Technological Forecasting and
Social Change (Frey & Osborne, 2013) suggests that 47% of
jobs in the United States are at risk of automation. Other
studies show that the workforce is already becoming more
automated (Acemoglu & Restrepo, 2017). Increases in
workplace automation bring an increased number of
interactions between humans and that automation. It is more
important than ever to understand and improve that
interaction.
Humans often fail to rely on automation properly (Lee &
See, 2000), displaying overconfidence in the system or
themselves, and trust calibration plays an important part in the
successful adoption of such automation (Lee & Moray, 1994;
Muir, 1987; Sheridan, 1999). Understanding and accurately
measuring trust in automated systems is the first step to
optimizing it, ensuring successful and helpful adoption:
“…Operators tend to use automation that they trust while
rejecting automation that they do not.” (Pop et al., 2015)
trust definitions and measures in HAI.
GOAL OF THIS RESEARCH
The goals of this research were to (1) survey current
literature for all useful measures of trust with respect to
human-automation interaction and (2) provide a formal record
for the quantity and types of research that utilize each method
of trust measurement.
METHOD
The following databases were searched for relevant
articles: PsycINFO (full list of covered journals can be found
at
https://www.apa.org/pubs/databases/psycinfo/coverage.aspx)
and PsycARTICLES (full list of covered journals can be found
at: https://www.apa.org/pubs/databases/psycarticles/coverage-
list.aspx). The chosen keywords for search were trust and
automation, and the systematic review was conducted in
October of 2018. The initial search was followed by a screen
of titles and abstracts to initially identify articles that did not
Proceedings of the Human
TRUST MEASUREMENT in Factors and Ergonomics INTERACTION:
HUMAN-AUTOMATION Society 2019 Annual Meeting REVIEW, San Jose State University
A SYSTEMATIC Page 2 of 1596
5

meet inclusion criteria. After the initial screen, papers that met Objective Measures. Objective measures varied greatly
inclusion criteria were reviewed in detail. between studies, especially between industries. Objective
Inclusion and Exclusion Criteria. Articles were included measures included compliance (n = 3), acceptance rate (n = 4),
if they met all inclusion criteria and excluded if they did not and adoption rate (n = 1). Other individually defined objective
meet at least one of the inclusion criteria. Inclusion criteria measures included resource sharing (defined as the proportion
were (1) the article reported research involving participants, of times a participant sought help from an automated decision
(2) the article was available for review in full-text, (3) the assistant) (n = 1), purchasing decisions (n = 1), glance rate in
article measured participant trust in addition to other automotive research (n = 1), and credits staked (defined as a
measurements (i.e. trust was not the only measure in the number of credits allocated to the automated system compared
study), and (4) the article measured human–automation, to credits kept by the user) (n = 1).
human–system, or human–computer trust where the primary
goal of the relevant system was to automate some function in Industry
the environment. Articles were organized into measures of
trust, industry, and type of automation. Articles were assigned to only one of the industries listed
In addition to the articles found in the search according to in Table 2. Out of interest, the researchers categorized articles
the criteria above, one article from external sources was by industry based on the claims made by the research and the
included for review. contextual information in each article using a grounded-theory
framework. This industry assignment resulted in the following
RESULTS distributions, organized by number of articles: automotive (n =
14), aviation (n = 6), military (n = 6), security (n = 6),
The initial search resulted in 217 articles. Following an medicine (n = 4), computer science (n = 3), manufacturing (n
initial screen, subsequent full-text review, and the addition of = 2), augmented reality (n = 1), maritime and nautical (n = 1),
the external article, 44 articles were found to meet all and unassigned (n = 1). The single unassigned article was a
inclusion criteria. study of visual screening with an automated decision assist
system with no contextual information concerning industry
Measures and was therefore not assigned to any of the industries listed
above.
Articles that met the inclusion criteria were screened for Industry Measures. Table 2 outlines the measures used by
the measure of trust employed in the research. All measures the reviewed articles as a function of the industry the article
were either marked as subjective or objective. Some studies was assigned to. For example, of the 14 studies in the
used only subjective measures, while others used only automotive industry, 11 of the 14 used subjective methods as
objective or both. The distributions are outlined in Table 1. the only measures of trust, and the remaining 3 used both
methods.
Table 1. Trust Measures by Type
Measure Articles Table 2. Trust Measures by Industry
subjective only 33 Articles by measure
objective only 5 Industry
both 6 subjective objective
both total
only only
Subjective Measures. Methods of trust measurement that automotive 11 3 14
were marked as “subjective” were self-reported ratings of the aviation 2 1 3 6
user’s trust in the automated system studied in each article. military 6 6
Articles measured trust in the system before interaction, security 5 1 6
during interaction, or after interaction. All articles using
medicine 2 2 4
subjective measures asked the participant to respond to one or
computer science 1 1 1 3
more questions about their trust in the system, following a
Likert scale (n = 34) or percentage (n = 4) and with a measure manufacturing 2 2
either developed alone by the researchers (n = 18) or augmented reality 1 1
following a model such as the scales below (n = 21). nautical 1 1
A total of 21 articles used the following scales to collect unassigned 1 1
self-reported trust information from their participants: The
Scale of Trust in Automated Systems (n = 8), the scale
developed by Muir (1989) (n = 8), the scale developed by Type of Automation
Merritt et al. (2011) (n = 2), the Human–Computer Trust
questionnaire (n = 2), or the German TiA Scale (n = 1). A total Using a grounded-theory approach, articles that met the
of 18 articles used methods defined individually by the inclusion criteria were also grouped by type of automated
researcher of each article. These methods are discussed in system that was used. These were assigned to 3 types of
more detail in the discussion section.
Proceedings of the Human
TRUST MEASUREMENT in Factors and Ergonomics INTERACTION:
HUMAN-AUTOMATION Society 2019 Annual Meeting REVIEW, San Jose State University
A SYSTEMATIC
automation shown in Table 3. Descriptions of each type of
automation are listed below in detail.
Table 3. Types of Automated Systems
Type Articles
decision aids 19
automated control and navigation 17
process control 8
Decision Aids. Automated systems whose primary
objective was to assist the user in making decisions about the
system, the environment, or the next step in a process were
assigned to the decision aids type. This type includes systems
such as automated navigation systems, automated alarm and
warning systems, maze solution systems, and more.
Many of the automated systems subjected to study in the
relevant articles focused primarily on image and video
processing (e.g. aerial target detection systems and X-ray
screening automation) (n = 8). These systems were classified
as decision aids because they met the criteria defined above.
Automated Control and Navigation. Automation systems
of this type were systems whose primary objective was to
automatically control the navigation of a vehicle or system
that the user would normally operate. These systems included
autonomous cruise control (ACC), automatic ground collision
avoidance systems (AGCAS), and other systems whose main
goal was navigation and were directly responsible for
providing control to the system/vehicle.
These systems are differentiated from decision aids
because the automated system itself was responsible for
controlling the vehicle or navigation path but allowed for
intervention. A navigation-focused decision aid simply
provided navigation information to the primary human
operator.
Process Control. Process control systems were automated
systems whose primary objective was automation of the
control and management of a given process. These included
automated process controls such as medication management
systems, simulated-world management agents, and air traffic
control automations.
Systems that were assigned to this type of automation
were differentiated from decision aids on the basis that they
were directly responsible for controlling the process system. A
process-focused decision aid allowed the human operator to
hold primary control of the process and consult the automated
system for aid in decision making.
After the review of relevant articles by automation type,
no relationship was observed between type of automation and
measure of trust other than what has already been stated. It is
worth noting that a majority of relevant research on automated
control and navigation systems belonged to articles assigned
to the automotive industry (n = 12).
DISCUSSION
Results showed that the current literature on trust in
human–automation interaction has a tendency to (1) use
mostly or only subjective measures of trust, (2) define and
Page 3 of 1597
5
measure trust in whatever manner investigator determines to
be useful to the system they are studying, and (3) be composed
of research in the automotive industry, with a proportionally
high number of automations focused on navigation assistance
or control.
Subjective Measures of Trust
A total of 75% of relevant articles (n = 33) used
subjective measures as the only method to collect trust
information in their research. A total of 14% of the remaining
articles (n = 6) used subjective methods in conjunction with
objective measures. Only 11% of relevant articles (n = 5) used
no subjective measures of trust whatsoever. In other words,
subjective measures are heavily relied on to measure trust. The
specifics of the measures referenced in the results are listed
below in greater detail.
Scale of Trust in Automated Systems. One of the two
scales that was most frequently used (n = 8) was The Scale of
Trust in Automated Systems developed by Jian et. al (2000).
This scale draws heavily on the work of Sheridan (1988), Lee
& Moray (1992), and Muir and Moray (1996). The work of
Jian, Bisantz, and Drury (2000) used three experiments to
configure a 12-item scale to measure trust in automated
systems and is clearly regarded as a source of authority in
measuring trust in automation. However, it is important to
note that:
“…the [Scale of Trust in Automated Systems] was
developed with respect to a nondirected feeling of trust in
automated systems, rather than trust in a specific system that
the participants had experienced.” (Jian et al., 2000)
This fact may have important implications for the validity
and accuracy of the scale when used to measure trust in a
specific automated system.
Muir Scale (1989). The scale referenced by many of the
articles (n = 8) and by a large body of research on trust in
automation, Lee and Moray (1992) used the trust measure
hypothesized by Muir (1989). This scale measures a user’s
trust in an automated system by asking 4 trust-focused
questions with answers on a scale of 1 (“not at all”) to 10
(“completely”).
Merritt et al. Scale (2011). The trust measure by Merritt
et al. (2011) is a 5-point Likert-type scale that assesses a
user’s trust in an automated system. This scale was used to
measure user trust in automated systems by a pair of relevant
articles (n = 2).
Human–Computer Trust. The Human–Computer Trust
(HCT) Questionnaire is a subjective measure of human–
computer trust developed by Madsen & Gregor (2000). The
HCT questionnaire is a 25-item subjective measure of
“cognition based” and “affective-based” trust. Madsen &
Gregor showed the scale has high internal consistency and
construct validity, and this scale was used by two relevant
articles (n = 2).
Proceedings of the Human
TRUST MEASUREMENT in Factors and Ergonomics INTERACTION:
HUMAN-AUTOMATION Society 2019 Annual Meeting REVIEW, San Jose State University
A SYSTEMATIC
German TiA Scale. The German TiA scale developed by
Körber et al. (2015) is a subjective measure of trust in
automated systems. It is a scale of 19 items on a Likert-type
rating scale with subscales for reliability and competence,
familiarity, trust, understanding, and intention of developers.
This scale has shown high internal consistency and construct
validity. It was used by one article (n = 1).
Other Measures. More than any other single measure,
41% of articles (n = 18) used scales and questionnaires
developed by the researchers themselves, not a scale
developed and validated by one of the authors above.
As stated above, 41% of relevant articles that used
subjective methods used methods defined individually instead
of scales that have been validated as measures of trust. While
individually developing a measure of trust may more closely
allow the investigators to understand the intricacies of the
system they are studying, their methods would need to be
validated externally, and even still, their measurements may
therefore not be generalizable to a system other than the one
they are studying. The practice of individually defining
measures of trust (1) brings to question the findings’
reliability, validity, and generalizability; and (2) creates a
precedent in the research of trust in human–automation
interaction that should be rejected.
With such a high reliance on subjective measures of trust,
researchers must commit to the use of reviewed and validated
scales. Many of the scales above show high internal
consistency and high construct validity, and are therefore
likely accurate measures of a user’s trust in the system, but
only provide value if they are used in research.
Industry
A majority of the relevant articles used subjective
measures of trust as the only method, but some industries were
more likely to have either both or objective-only measures.
For example, 21% of automotive articles (n = 3) and 50% of
aviation research used both methods (n = 3). As expressed in
the relevant articles, the use of objective measures may be
more useful or necessary depending on the industry; for
example, a regular practice in research of the automotive
industry is to track eye movements or the duration that drivers
attend to a particular area of their visual field. Applied to
testing new automated driving systems, this eye-tracking
information can provide researchers with information about
how often a user “checks on” the status of the system, an
objective measure of their trust in that automation.
Table 2 presents each industry in order of the number of
relevant articles. Automation was by far the most prevalent
industry (n = 14), followed by aviation (n = 6), military (n =
6), and security (n = 6). This shows a prevalence of research
of trust in human–automation interaction from the automotive
industry and may be an indication of the important role of trust
in automated systems from that industry.
Page 4 of 1598
5
Automation Type
The type of automation that each study researched plays a
large role in defining the type of trust measure used and its
generalizability to other areas. Decision aid systems were the
most researched type of automation (n = 19), followed by
automated control and navigation systems (n = 17).
While decision-aid research was conducted across
industries, a majority of automated control and navigation
systems were from the automotive. This impacted the
measures of trust that each collected. Specifically, automated
control and navigation systems used glance rates, monitoring
rates, and compliance as their objective measures of trust.
These measures may not be important or accurate measures of
trust in other industries.
The distinction between measure, industry, and type of
automation is not intended to make claims about how
prevalent automation is in each industry but simply provide a
foundation for which to present the relevant research in each
of the categories. All of the listed industries will be subject to
automation in the near future, so it is important for all to
understand and optimize trust in automation.
CONCLUSIONS
As evidenced by the results of this systematic review,
measures of trust vary across industry, type of automated
system, and by study. Although frequency counts are a useful
measure of current state-of-the-art, they should not be
interpreted as an indicator of the best method to use, just what
is being used.
Validated and peer-reviewed subjective measures of trust
like the scales developed by Muir and Merritt and the HCT
questionnaire provide an accurate picture of a user’s trust in
the system, but those scales are not used frequently enough.
To ensure accurate trust measurement in HAI, the highest
priority for researchers may be to use a validated scale for
subjective measurement of trust and then combined with other,
more-system-specific measures if necessary.
The current literature on trust in HAI suggests that with
the high internal consistency and construct validity of verified
subjective measures, such as the Muir and Merritt scales, it
may not be necessary or even prudent to mandate that trust
measurement includes objective methods as long as requisite
care is taken in maintaining validity of the scales; if users are
able to accurately predict their level of trust by objective
measure in a given automated system, it may not be necessary
to measure their trust objectively. If participants’ trust is
measured with only objective measures, the investigators may
need to prove the validity of these measures, and the results of
their research may be able to be generalized less.
Across industry and system type, the use of peer-
reviewed, verified, and effective measures of trust in human–
automation interaction is of the utmost importance. It may not
be preferable to standardize a single measure for all
applications, but maintaining consistency among industry, or
at least among scientific research, is critical for building a
Proceedings of the Human
TRUST MEASUREMENT in Factors and Ergonomics INTERACTION:
HUMAN-AUTOMATION Society 2019 Annual Meeting REVIEW, San Jose State University
A SYSTEMATIC
viable basis for future research and for safe, usable, and
satisfying human–automation interaction.
Validating Trust Measures. Measure validation is another
issue of importance for standardizing trust across industries.
Measures such as the HCT questionnaire and German TiA
scale have been validated but only in the context of current
human–automation interaction. These measures inherently
were created with exact applications in mind and are validated
as measures of trust in specific ways. The variety of
automation in industry makes standardizing validation of these
measures difficult. Industries, by definition, have very
different applications of automated technology, and with these
come the necessity of validating measures differently. For this
reason, it may be important to validate these scales across
industry and with new systems of automation.
Limitations
The most noteworthy limitation of this review is the
limitations of the relevant articles themselves. Participants of
most relevant research had collegiate-level education and
significant exposure to computers and automated systems.
This undoubtedly has implications for self-reporting trust in
automated systems; with training and prolonged exposure to
automation, the participants may have a more-accurate sense
of how much they trust the system or may be more inclined to
trust the system in general, skewing the validity of subjective
measures of trust.
Reviewers were also restricted in the following ways: (1)
reviewers had limited access to relevant databases, (2)
reviewers had limited access to pertinent articles that did not
match the defined keywords, (3) many articles could not
accessed, (4) applicable articles that were published outside of
the United States or were not in the English language did not
receive proportional review, and (5) research being conducted
at the time of the review or after the review was conducted
could not be included.
Implications for Future Research
The discipline of human-robot trust was not studied but is
known to have models of trust that may be able to be
translated to HAI. Subjective measures such as the 14-point
Human Robot Trust Scale, Schaefer (2013) may be able to be
translated to HAI. Importantly, objective, quantitative
methods, such as using confirmation change as done by
Gaudiello, Zibetti, Lefort, Chetouani (2016) is a powerful tool
that should be examined in the context of HAI as well.
With the widespread, and recommended, use of subjective
measures of trust in human–automation interaction comes the
necessity to continue to research and validate the efficacy of
these measures as automation continues to evolve. Even with
high internal consistency and construct validity, measures are
only useful if they apply to the tools and systems they are
being used with. As systems change and human–automation
interaction becomes more prevalent, these tools will need to
be re-validated and improved for future research.
Page 5 of 1599
5
REFERENCES
Acemoglu, D., & Restrepo, P. (2017). Robots and Jobs: Evidence from US
Labor Markets. National Bureau of Economic Research (NBER).
doi:10.3386/w23285
Alexander, V., Blinder, C., & Zak, P. J. (2018). Why trust an algorithm?
Performance, cognition, and neurophysiology. Computers in
Human Behavior, 89, 279–288. doi:10.1016/j.chb.2018.07.026
Balfe, N., Sharples, S., & Wilson, J. R. (2018). Understanding is key: An
analysis of factors pertaining to trust in a real-world automation
system. Human Factors, 60(4), 477–495.
doi:10.1177/0018720818761256
Crowley, R.S., Legowski, E., Medvedeva, O. et al. (2013). Automated
detection of heuristics and biases among pathologists in a
computer-based system. Adv in Health Sciences Educ, 18(3), 343-
363. doi:10.1007/s10459-012-9374-z
Frey, C. B., & Osborne, M. A. (2017). The future of employment: How
susceptible are jobs to computerisation? Technological Forecasting
and Social Change, 114, 254-280.
doi:10.1016/j.techfore.2016.08.019
Gaudiello, I., Zibetti, E., Lefort, S., Chetouani, M., & Ivaldi, S. (2016). Trust
as indicator of robot functional and social acceptance. An
experimental study on user conformation to iCub answers.
Computers in Human Behavior, 61, 633-655.
Hoff, K. A., & Bashir, M. (2015). Trust in Automation: Integrating Empirical
Evidence on Factors That Influence Trust. Human Factors, 57(3),
407–434. doi:10.1177/0018720814547570
Jian, J.-Y., Bisantz, A. M., & Drury, C. G. (2000). Foundations for an
empirically determined scale of trust in automated systems.
International Journal of Cognitive Ergonomics, 4(1), 53–71.
doi:10.1207/S15327566IJCE0401
Körber, M., Bengler, K. (2014). Potential individual differences regarding
automation € effects in automated driving. In: Gonzalez, C.S.G.,
Ord onez, C.C., Fardoun, H. ~ (Eds.), Interaccion 2014:
Proceedings of the XV International Conference on Human
Computer Interaction. ACM, New York, NY, USA, pp. 152e158.
doi:10.1145/2662253.2662275.
Lee, J. D. (1991). The dynamics of trust in a supervisory control simulation.
In Proceedings of the Human Factors Society 35th annual meeting
(pp. 1228–1232). Santa Monica, CA: Human Factors Society
Lee, J., & Moray, N. (1992). Trust, control strategies and allocation of
function in human-machine systems. Ergonomics, 35(10), 1243–
1270. doi:10.1080/00140139208967392
Lee, J. D., & Moray, N. (1994). Trust, self-confidence, and operators’
adaptation to automation. International Journal of Human-
Computer Studies, 40(1), 153–184. https://doi-
org.libaccess.sjlibrary.org/10.1006/ijhc.1994.1007
Lee, J. D., & See, K. A. (2004). Trust in Automation: Designing for
Appropriate Reliance. Human Factors, 46(1), 50–80. https://doi-
org.libaccess.sjlibrary.org/10.1518/hfes.46.1.50.30392
Muir, B. M. (1987). Trust between humans and machines, and the design of
decision aids. International Journal of Man-Machine Studies,
27(5/6), 527–539.
Muir, B. M. (1987). Trust in automation: Part II. Experimental studies of trust
and human intervention in a process control simulation.
Ergonomics. 39, 429-460.
Muir, B. M. (1989). Operators' trust in and percentage of time spent using the
automatic controllers in a supervisory process control task.
Doctoral thesis. University of Toronto.
Madsen, M., Gregor, S. (2000). Measuring human-computer trust. In:
Proceedings of 11th Australasian Conference on Information
Systems. 6e8 December. Australasian Association for Information
Systems. QUT, Brisbane, pp. 6e8.
Schaefer, K. E. (2013). The perception and measurement of human-robot trust
(Doctoral dissertation, University of Central Florida Orlando,
Florida).
Sheridan, T. B. (1988). Trustworthiness of command and control systems.
Proceedings of the Third IFAC/IFIP/IEA/IFORS Conference on
Man Machine Systems (pp. 427-431). Elmsford, NY: Pergamon.
Sheridan, T. B. (1999). Human supervisory control. In A. P. Sage & W. B.
Rouse (Eds.), Handbook of systems engineering and management
(pp. 645–690). New York, NY: Wiley & Sons.