875376 HFSXXX10.

1177/0018720819875376Human FactorsEvaluation of Objective Measures of SAresearch-article2019

A Systematic Review and Meta-Analysis of Direct

Objective Measures of Situation Awareness:
A Comparison of SAGAT and SPAM
Mica R. Endsley , SA Technologies Inc., Mesa, Arizona, USA

Objective: To examine evidence of sensitivity, pre- Introduction

dictiveness, and methodological concerns regarding direct,
objective measures of situation awareness (SA). The study of situation awareness (SA) has
Background: The ability to objectively measure SA grown over the past three decades to encompass
is important to the evaluation of user interfaces and dis- investigations of not only the construct, but also
plays, training programs, and automation initiatives, as well the evaluation of new system designs and train-
as for studies that seek to better understand SA in both
individuals and teams. A number of methodological criti-
ing programs in a wide variety of domains. To
cisms have been raised creating significant confusion in the accommodate these efforts, the need to measure
research field. SA has remained fundamental to progress in
Method: A meta-analysis of 243 studies was conducted this area of research. A number of approaches
to examine evidence of sensitivity and predictiveness, and have been proposed and used in research on
to address methodological questions regarding Situation
Awareness Global Assessment Technique (SAGAT), Situ-
SA, including process measures, performance
ation Present Assessment Technique (SPAM), and their measures, and measures that seek to assess a
variants. person’s level of knowledge and understanding
Results: SAGAT and SPAM were found to be equally about the situation via direct questioning of the
predictive of performance. SPAM (64%) and real-time individual. These techniques are summarized
probes (73%) were found to have significantly lower sensi-
tivity in comparison to SAGAT (94%). While SAGAT was
in Table 1, along with some advantages and
found not to be overly memory reliant nor intrusive into disadvantages of each. See Endsley (1995b) or
operator performance, SPAM resulted in problems with Endsley and Jones (2012) for a review. SA mea-
intrusiveness in 40% of the studies examined, as well as surement approaches will be briefly described,
problems with speed-accuracy tradeoffs, sampling bias, followed by a focus on the direct measurement
and confounds with workload. Concerns about memory
reliance, the utility of these measures for assessing Team
of SA as an ongoing state of knowledge about
SA, and other issues are also addressed. relevant information in the environment that is
Conclusion: SAGAT was found to be a highly sensi- needed to support decision-making in complex
tive, reliable, and predictive measure of SA that is use- and dynamic domains.
ful across a wide variety of domains and experimental Some researchers have examined the pro-
settings.
Application: Direct, objective SA measurement pro-
cesses that people use to develop SA using mea-
vides useful and diagnostic insights for research and design sures such as eye tracking (de Winter, Eisma,
in a wide variety of domains and study objectives. Cabrall, Hancock, & Stanton, 2019; Ikuma, Har-
vey, Taylor, & Handal, 2014; Smolensky, 1993),
Keywords: situation awareness, metrics, SAGAT, SPAM, communications (Bolstad et al., 2007; Gorman,
measurement Cooke, Pederson, Connor, & DeJoode, 2005;
Orasanu, 2000; Prince, Salas, & Stout, 1995),
verbal protocols (Hall & Phelps, 1983; Rose,
Address correspondence to Mica R. Endsley, SA Bearman, Naweed, & Dorrian, 2019; Sullivan &
Technologies, 5301 S. Superstition Mountain Drive, Suite
104377, Gold Canyon, AZ 85118, USA; e-mail: mica@
Blackman, 1991; Walker, Stanton, & Young,
satechnologies.com. 2008), and physiological measurement (Wilson,
2000). While process measures can provide
HUMAN FACTORS
Vol. XX, No. X, Month XXXX, pp. 1­–27
insights into how people develop SA, they can
DOI: 10.1177/0018720819875376 only be used to indirectly infer the quality and
Article reuse guidelines: sagepub.com/journals-permissions completeness of the resulting SA, as a state of
Copyright © 2019, Human Factors and Ergonomics Society. knowledge about the situation, obtained by the
2
Table 1: Comparison of SA Measurement Approaches
Process Measures
Metrics
  Eye Tracking, Communications,
Verbal Protocols, Physiological
Advantages
•• Objective and continuous
•• Eye tracking provides information
on order and duration of attention
to visual information
•• Communications and verbal
protocols can provide information
on processes, strategies, types of
assessments made
Disadvantages
•• Eye tracking does not capture
attention to auditory cues or
if information is correctly
understood or integrated for
higher levels of SA
•• Communications provides only
partial information on what is
attended to and how processed.
Some people verbalize more than
others.
•• Little research to date to support
validity of physiological measures
for SA
Note. SA = situation awareness; SART = Situational Awareness Rating Technique; SAGAT = Situation Awareness Global Assessment Technique; SPAM = Situation
Present Assessment Technique.
Performance Measures
Response time, Errors
•• Objective or subjective
•• Can be gathered without
operator input
•• Often already collected
•• Assumes what behavior will
occur given a particular state
of SA. System or training
changes may affect
performance in unexpected
ways.
•• SA for normal and
emergency events may
be different, so inferences
constrained by scenarios
tested and performance
measures collected
•• Confuses SA with
performance which can be
affected by other factors.
Often insufficient sensitivity
and diagnosticity.
Direct SA Measures
Subjective Objective
Likert-type scales, SART SAGAT
•• Easy to collect •• Queries people on
relevant SA knowledge
on perceptions,
comprehension, and
projection
•• Can be used across •• Objectively scored based
many domains on simulation data
•• Provides indication •• Unbiased sampling
of confidence in SA across scenario avoids
end-of-trial memory
dependence
•• People may not be •• Requires freezing of
aware of what they simulation scenario
do not know; meta-
awareness is poor
•• May be overly •• Requires people to answer
influenced by self queries based on memory
assessments of for 2–3 min during freeze
performance
•• Some scales (e.g., •• Requires development
SART) include of domain-specific queries
measures of
workload
SPAM
•• Queries people on relevant
SA knowledge of past,
present, and future
•• Objectively scored and timed
based on simulation data
•• Simulation freeze not
required
•• Requires dual-tasking
to answer queries while
performing task, potentially
interfering with performance
and creating a secondary
task workload measure
•• Allows people to “look-up”
answers to queries which
may not assess SA
•• Requires development of
domain-specific queries
Evaluation of Objective Measures of SA 3
individuals involved. An individual’s knowl-
edge and capabilities, as well as the system
interfaces available, mediate the degree to which
the SA processes used are successful in creating
accurate SA; so even two people using the same
process may not arrive at the same situation
understanding. Furthermore, these techniques
generally provide only partial insights into what
information is attended to and how it is pro-
cessed to form situation comprehension and pro-
jection. Verbal protocols have not been found to
be effective for measuring SA (Rose et al., 2019;
Walker et al., 2008), and physiological measures
have had little research to establish validity for
this purpose. Therefore, process measures are
inadequate for measuring SA on their own.
Other researchers have focused on trying to
infer whether or not people have good SA based
on their actions and performance; however, this
approach can be severely hampered when peo-
ple do not act as expected (Farley, Hansman,
Amonlirdviman, & Endsley, 2000; Jones &
Endsley, 2000b; Pritchett, Hansman, & Johnson,
1995). While collecting data on performance is
always desirable, the main limitation of this
class of measures is that it only infers SA indi-
rectly, in what can be a circular argument, with-
out providing sufficient detail or diagnosticity
on what the operator really thought was going
on. In addition, Wickens (2000) pointed out that
SA for normal and emergency events may be
quite different, thus inferences about SA are
highly constrained by the scenarios that are
tested. In that many other factors can effect the
decisions and actions people make based on
their SA (such as tactics, procedures, and execu-
tion skills), behavioral and performance mea-
sures also offer only limited insights into SA.
Largely due to the limitations of the above
approaches, the majority of research employing
SA measures has focused on directly determin-
ing a person’s SA, as a state of knowledge,
through either subjective or objective assess-
ment (Endsley & Garland, 2000). Subjective
measures of SA are easy to obtain; however,
they suffer from the fact that people may be
unaware of what they do not know and may be
unduly influenced by observations of perfor-
mance (Endsley, 1995b; Endsley & Jones,
2012). Research indicates that subjective SA
ratings more likely assess an individual’s confi-
dence level (Endsley, Selcon, Hardiman, &
Croft, 1998; Hamilton, Mancuso, Mohammed,
Tesler, & McNeese, 2017; Sulistyawati, Wick-
ens, & Chui, 2009). Observer ratings of SA
overcome some problems with self-assessment,
but still suffer from becoming a proxy for sub-
jective performance ratings in that observers
have limited information on a person’s mental
representation of the situation. See also Endsley
(in press).
Objective SA measures assess the operator’s
knowledge of the current situation by asking the
operator relevant questions about the situation
that can be objectively scored as correct or incor-
rect. Two commonly used metrics in this cate-
gory are the Situation Awareness Global Assess-
ment Technique (SAGAT) (Endsley, 1988,
1995b) and the Situation Present Assessment
Technique (SPAM) (Durso et al., 1998). How-
ever, these methods have been the subject of
considerable debate, which has created confu-
sion in the field (Chiappe, Rorie, Moran, & Vu,
2012; Durso et al., 1998; Endsley, 1995a; Jones
& Endsley, 2004; Salmon et al., 2008; Salmon,
Stanton, & Young, 2011; Sarter & Woods, 1991).
The objective of this article is to examine the
SAGAT and SPAM methods for objective mea-
surement of SA and to review and compare the
success of these metrics. I first provide an over-
view of these two SA measurement approaches
along with a summary of their advantages and
disadvantages. I then focus on questions of sen-
sitivity and predictiveness, performing a meta-
analysis of the research literature that has been
conducted on and with these SA metrics. In
addition, a systematic review of the available lit-
erature is conducted to address a number of
methodological issues that have been raised
concerning the construct validity of the two
techniques.
SAGAT
SAGAT is one of the earliest and most
widely used measures of SA. When using
SAGAT, simulations of representative tasks or
scenarios are frozen at randomly selected times,
and system displays are blanked while people
quickly answer questions about their current
perceptions of the situation. Queries can be
4 Month XXXX - Human Factors
provided verbally, via pencil and paper, or on
a computer or tablet for ease of administration.
SAGAT queries correspond to an individual’s
SA requirements as determined from the results
of a SA requirements’ analysis, such as a Goal
Directed Task Analysis (Endsley, 1993b; End-
sley & Jones, 2012), for a given domain and
role. People’s perceptions are then compared to
the real situation, based on simulation computer
databases, to provide an objective measure of
SA. Scoring some queries, such as those related
to situation comprehension, may be provided by
subject matter experts with perfect knowledge
of the situation at the time of the freeze. Multi-
ple “snapshots” of a person’s SA are acquired in
this way, providing an assessment of the quality
of SA provided by a particular system design.
SAGAT provides an objective, unbiased
assessment of SA by providing the queries at ran-
dom times across a scenario, collecting data dur-
ing both high-workload and low-workload peri-
ods. By assessing SA at different points in time,
SAGAT avoids the problem of relying on mem-
ory of events after the scenario is completed
(Nisbett & Wilson, 1977). As a global measure-
ment tool, SAGAT includes queries across a
wide range of SA requirements for a given job,
including Level 1 (perception of data), Level 2
(comprehension of meaning), and Level 3 (pro-
jection of the near future). This includes a con-
sideration of system functioning and status, as
well as relevant features of the external environ-
ment and team as appropriate. Since SAGAT
includes queries across the full spectrum of an
individual’s SA requirements, this approach min-
imizes the possible biasing of attention, as people
cannot prepare for the queries in advance (Ends-
ley, 1995b). Examples of queries for air traffic
control (ATC) are shown in Table 2.
SAGAT scores are normally expressed as
percent correct for each query, based on opera-
tionally relevant tolerance bands. Many re-
searchers have varied from this recommended
approach, instead combining the scores on all
SAGAT queries into a combined overall score,
or into three combined scores that represent
Level 1, Level 2, and Level 3 SA. Other scoring
variants include (a) the SA Control Room
Inventory (SACRI), which computes a d′ sensi-
tivity and bias score based on signal detection
theory (Hogg, Follesø, Strand-Volden, & Tor-
ralba, 1995); (b) the Qualitative Assessment of
SA (QUASA), which poses probes in the form
of true/false statements and adds an assessment
of confidence in each answer (McGuinness,
2004); (c) SALSA, which attempts to weight
each query type in computing an overall score
(Hauss & Eyferth, 2003); and (d) the SA Verifi-
cation and Analysis Tool (SAVANT), which
provides a partial display to ask questions about
missing information, and includes an assess-
ment of time to answer each question along
with accuracy (Willems & Heiney, 2001).
SPAM
SPAM and real-time probes also provide
queries to assess SA; however, the queries are
provided in real time, usually verbally, while
the individual is carrying out his or her normal
operational tasks. In addition to response accu-
racy, the time to respond to each SA probe is
collected as an index of how readily available
the information is. This form of measurement
is referred to as a real-time probe (Endsley,
1995b; Jones & Endsley, 2000a). SPAM also
provides a “ready” prompt prior to each SA
probe that allows operators to delay receiving
it until they are ready for the probe. SA probes
may be staged as an embedded probe, with a
confederate playing the role of the questioner,
so that they appear to be natural to the scenario.
SPAM provides queries that correspond to past,
present, and future aspects of the situation.
Methodological Issues and Concerns
A number of studies have attempted to
directly compare SAGAT and SPAM, focusing
on their relative abilities to predict performance
in the simulation, with differing conclusions
(Durso, Bleckley, & Dattel, 2006; Durso et al.,
1998; Jones & Endsley, 2004; Loft, Bowden,
et al., 2015; Pierce, Strybel, & Vu, 2008; Stry-
bel, Vu, Kraft, & Minakata, 2008). In addition,
many researchers have sought to determine how
sensitive these techniques are to the independent
manipulation provided in the experiment (Alex-
ander & Wickens, 2005; Jones & Endsley, 2004;
Silva, Grigoleit, Ann Burress, & Fitzpatrick,
2017; Vidulich, 2000). A goal of the present
Evaluation of Objective Measures of SA 5

Table 2: Sample SAGAT Queries for Air Traffic Control

Level 1 SA
  Mark all aircraft on the attached sector map.
    (Completed map provided for all subsequent questions)
  What is the airspeed of the indicated aircraft?
  What is the heading of the indicated aircraft?
  What is the type of the indicated aircraft?
  Is the indicated aircraft currently level, climbing, or descending?
  Which aircraft are currently experiencing an emergency?
Level 2 SA
  Which aircraft have been issued assignments (clearances) that have not yet been completed?
  Which aircraft are not conforming to their clearance?
  Which aircraft are not in communication with you?
  Which aircraft are currently being affected by weather?
  Which aircraft will violate special airspace separation standards if they stay on their path?
Level 3 SA
  What is the next sector of the indicated aircraft?
  Which pairs of aircraft will lose separation if they stay on their current (assigned) course?
  Which aircraft must be handed off to another sector/facility in the next 2 min?

Note. SAGAT = Situation Awareness Global Assessment Technique; SA = situation awareness.

study is to examine the existing research base
to address these questions and to determine how
well these two approaches to objective SA mea-
surement fare in terms of sensitivity and predic-
tive ability. Rather than making an assessment
based on any one study, a meta-analysis across
the available research literature is conducted to
provide a clearer picture of the utility and pre-
dictive validity of these two metrics.
The research literature is also rife with a
number of criticisms of SAGAT and SPAM that
need to be addressed. First, a number of research-
ers have criticized SAGAT claiming that the
freezes to collect data are intrusive and that it
relies too much on working memory (Chiappe
et al., 2012; Durso et al., 1998; Salmon et al.,
2011; Sarter & Woods, 1991). They advocate for
the SPAM approach, claiming that it produces a
picture of SA that is more natural and “situated”
(Chiappe, Strybel, & Vu, 2015).
On the other hand, concerns have also been
raised about the intrusiveness of real–time probes
and SPAM, in that they require people to multi-
task to answer questions while performing oper-
ational tasks, which could negatively affect pri-
mary task performance (Endsley, 1995b; Jones &
Endsley, 2000a; Pierce, 2012). Because of this,
SPAM could actually be considered a secondary
workload task, even though operators are allowed
to delay answering until they feel able. The abil-
ity to wait to answer probes until the participant
is ready also creates a problem, as it systemati-
cally biases SPAM results toward lower work-
load periods. Another concern is that the ability
to look at displays while answering questions
fails to capture SA as an ongoing understanding
of the world, but rather measures people’s ability
to look up information (Endsley, 2015a).
The literature base is full of these logical, but
differing, viewpoints, and many of the claims
have been offered without evidence. Therefore,
empirical research addressing these method-
ological issues will be examined, including the
potential intrusiveness of the measures, their
reliance on working memory, speed accuracy
trade-offs that may affect outcomes, sampling
bias, potential confounds with workload, and
query design. The utility of these measures for
assessing Team SA is also considered.
Method
A literature search was conducted on Google
Scholar, Scopus, and Science Direct with the
key words SA, probes, SAGAT, SPAM, SACRI,
QUASA, SAVANT, and SALSA. Papers that
collected experimental data with one of these
techniques (or variants of them), with sufficient
data reported, were included in this review.
6 Month XXXX - Human Factors
Duplicates of the same study were excluded
where identified. A total of 243 papers were tab-
ulated that include data on metrics, experimental
environment, domain area, type of participants,
number of participants, and experimental results
of the study, including statistical analyses.
Given that SACRI, QUASA, SAVANT, and
SALSA are all variants of SAGAT, based on the
same freeze and query technique, these measures
are reported on together. Differences in results
associated with the effect of these different anal-
ysis approaches are then broken out in the meta-
analysis. Similarly, both real-time probes and
SPAM are based on the same technique of pro-
viding probes to operators during task perfor-
mance, although SPAM also provides a ready
prompt prior to probe administration. These two
techniques are also reported on together; how-
ever, differences in findings for these two varia-
tions are broken-out in the meta-analysis.
Sensitivity
To determine the sensitivity of a measure,
typically an attempt is made to independently
vary the construct of interest and to assess
how well the measure detects this change. In
that there is no way to independently vary SA
directly, the best proxy in the SA research lit-
erature is the variation provided by the experi-
mental manipulation in the study (e.g., changes
in displays, automation, training condition, or
operator experience). The sensitivity meta-
analysis, therefore, examined the sensitivity of
the measures to the experimental manipulation
in each study, that is, the likelihood that the
measures would find a difference in SA between
conditions, assuming that true differences exist.
While it is almost impossible to say that every
study manipulation should result in an effect
on SA, the hypothesis of each study was that
such a difference should theoretically be found.
Furthermore, when performance or workload
measures in the same study found differences
between conditions, this increases the expecta-
tion that the SA measures should be sensitive to
study manipulations.
Other approaches have also been applied to
examining the sensitivity of metrics in meta-
analyses, including Cohen’s d (Cohen, 1995),
Cohen’s h (Cohen, 1995), and Cramer’s V
(Cramer, 1946), which determine comparative
effects sizes. Unfortunately, calculating the
comparative sensitivity of the SA measures
using any of these approaches was not possible,
due to the differing types of output measures
provided by the SA metrics. Cohen’s d is often
used to compare relative differences between the
means of two different groups. So Cohen’s d
could be used to show the effects size of a mea-
sure such as SPAM that produces response time
as its outcome measure. However, this metric is
inappropriate for measures where the outcome is
expressed as a proportion (i.e., percent correct),
such as SAGAT. In such a case, Cohen’s h is
more appropriate. Furthermore, some of the
studies examined provided only chi-square sta-
tistics for frequency counts, in which the appro-
priate metrics for an effects size calculation is
Cramer’s V. Therefore, it was not possible to
rely on the calculation of effect sizes in each
study as a common method for comparison
across these different types of SA measures,
since these analyses rely on different statistics
that are not comparable.
Instead, sensitivity of each measure was cal-
culated as the percentage of studies where the
metric was employed in which it detected a dif-
ference between study conditions (i.e., probabil-
ity of detection). This approach has been used by
Vidulich (2000) and Bushman (1994), and is
based on considering the independent variable
in the study to have been a means of manipulat-
ing SA. The sensitivity of the metric can then be
assessed as its ability to detect that change. As
Wickens (1998) has argued in favor of consider-
ing larger p-values than .05, the analysis also
considered studies where the metrics showed a
significant difference between conditions at a
p-value of up to .10. If the measure was found
sensitive (p < .05) it received a score of 1, if it
was marginally significant (p < .10) it received a
score of .5, and if it was not significant (p > .10)
it received a score of 0. This provided for an
assessment of the proportion of studies using
each metric that showed sensitivity to the stud-
ies’ manipulation of SA. The relative sensitivity
of the SA metrics across different domains, test
environments, subject types, and methodologi-
cal differences were also compared to determine
if these factors were relevant to metric sensitiv-
ity. In addition, studies that employed both
SAGAT and SPAM in the same study were
Evaluation of Objective Measures of SA 7
analyzed separately to determine relative sensi-
tivity between the measures.
Predictiveness
Another issue of concern has been whether
the SA measure employed is predictive of per-
formance and if so, whether SAGAT or SPAM
is more predictive than the other. Assessing the
predictiveness of SA metrics against perfor-
mance measures is rather complicated. First, it
assumes that SA metrics should be predictive of
performance. While this is true at a high level, it
neglects the fact that both SA and performance
are multi-dimensional. For example, Wickens
(1995) shows that the SA needed for routine
performance may not be the same as that needed
for emergency performance. And SA of some
information may only be relevant to some per-
formance outcomes. For example, knowing how
fast the driver of a car is going may be relevant
for getting speeding tickets, but not for lane
tracking accuracy. Not all SA is relevant to all
performance measures. Furthermore, most stud-
ies are limited in the number of performance
measures assessed, increasing the likelihood
that some SA metrics may not have the relevant
performance metrics for comparison. Therefore,
this meta-analysis assesses whether any SA
measure was predictive of any performance
measure in each study.
Following the same approach as the sensitiv-
ity meta-analysis, predictiveness was calculated
as the proportion of studies using a metric in
which the SA measure was predictive of at least
one performance measure. If the measure was
found to be predictive (p < .05) of performance
in the study, it received a score of 1, if it was
marginally predictive (p < .10) it received a
score of 0.5, and if it was not predictive it was
assigned a score of 0.
In addition, the degree of strength of the cor-
relations between SA measures and the perfor-
mance measures across studies was also calcu-
lated, by converting each Pearson’s r to Fisher’s
Z. This provided for a direct comparison between
the mean correlation strength (performance pre-
dictiveness) and a 95% confidence interval
across metrics. The relative predictiveness of the
SA metrics across different domains, test envi-
ronments, subject types, and methodological
differences were directly compared to see if
these factors affected predictiveness. Papers that
employed both SAGAT and SPAM in the same
study were analyzed separately to determine
relative predictiveness of the measures.
Intrusiveness
Studies that directly compared performance
and or workload in trials in which the SA
measures were employed to trials that did
not include a SA measure were analyzed to
determine evidence of the intrusiveness of the
techniques. Intrusiveness was determined by
whether the SA measure significantly (p < .05)
affected a performance metric in the study.
Memory Dependence
Whether or not SAGAT is overly reliant on
working memory was analyzed by qualitatively
examining the studies that also independently
assessed working memory to determine the rela-
tionship between these two measures. Differ-
ences between expert and novice subjects were
considered. Because SPAM allows operators to
see all information when answering queries, it
was not considered in this analysis.
Speed Accuracy Trade-offs
As the predominant measure for SPAM is the
time to respond to the SA query, this analysis
focused on studies that provided evidence of
speed-accuracy trade-offs that potentially con-
found this measure. Because SAGAT does not
involve timed performance, but only accuracy,
it was not the focus of this analysis.
Sampling Bias
Both SAGAT and SPAM should provide an
unbiased estimate of SA by sampling operator
knowledge across a wide range of conditions
and events during a scenario. Studies were
examined to determine whether the provision
of a “ready” prompt with the SPAM technique,
allowing people to defer answering, created a
bias toward lower workload periods.
SA vs. Workload
Studies that examined the relationship
between the SA measures and workload were
8 Month XXXX - Human Factors
also assessed to determine whether the measures
assess SA as an independent construct, or are
confounded by workload.
Query Design
Differences in the content and nature of the
SA queries provided in SAGAT and SPAM
studies were qualitatively assessed.
Team SA
The SA of teams (as an aggregate or in terms
of the shared SA among team members) is
often a focus of research interest. Studies that
used either SAGAT or SPAM to measure team
SA were analyzed separately to determine the
degree to which these techniques support SA
research at the team level in addition to at the
individual level.
Results and Discussion
The presentation of results and discussion
will be considered together for each aspect of
the meta-analysis: sensitivity of the metrics
to the study manipulation, and predictiveness
of the metrics for the performance measures
provided in the study. This is followed by a
discussion of findings with regard to intrusive-
ness of the techniques, memory dependence,
the presence of speed-accuracy tradeoffs, the
potential for sampling bias, distinguishability
from measures of workload, query design, and
ability to measure team SA.
Sensitivity
In all, 171 papers were found that used a
direct, objective SA measure to evaluate a
system design (hardware or displays), automa-
tion, operational concept, attention allocation,
training manipulation, or to assess differences
in participants and the effects of other factors
such as workload. Of those, 150 papers involved
SAGAT or a variant, and 27 involved SPAM or
a real-time probe (with 6 papers including both
measures).
SAGAT and variants.  Of the 150 papers that
included SAGAT or a variant of it, 5 papers were
duplicates, 1 paper included 3 studies, and 5
papers included 2 studies, making for 152 unique
studies in all. These papers are listed in Appendix
A, which is available with the manuscript on the
Human Factors website. The proportion of stud-
ies in which SAGAT or one of its variants was
sensitive to the experimental manipulation in the
study is listed in Table 3. The sensitivity score
across all 152 studies is 85.5%.
Studies were conducted in a number of
domains including aviation, ATC, driving, mili-
tary, medical (including nursing, surgery, and
other specialty areas), process control, robotics
and unmanned air vehicle control, artificial tasks
created specifically for experiments, and other
areas including firefighting (4), train driving (2),
maritime (2), teleoperations (2), emergency
management, weather forecasting, cyber secu-
rity, and maintenance. Overall, sensitivity of the
measure was not significantly different across
domains (χ2 = 2.47, p = .96).
The majority of studies were conducted in
simulations or microworlds, but there were also
studies that employed the viewing of videos and
live exercises. There were no significant differ-
ences (χ2 = 0.38, p = .94) in the sensitivity of the
technique between testing environments, nor in
whether the test subjects were experienced
domain practitioners, novices in the domain, or
were completely naïve (either students or from
the general population) (χ2 = 2.26, p = .69).
Significant differences in sensitivity were
found based on how the measure was adminis-
tered. Studies that only measured SA at the end
of trials (73% sensitivity) experienced lower
sensitivity than those using the freeze technique
prescribed by SAGAT (89% sensitivity), (z =
2.02, p = .02). Studies that collected SA only at
the end of trials generally suffered from a low
sample size as well as only capturing SA at the
end of the trial, which may not be indicative of
SA throughout.
A second source of variance occurred due to
the method of scoring used. The SAGAT meth-
odology recommends scoring each query sepa-
rately because many design interventions can
effect changes in just one or two elements of SA,
often in different directions, and if the queries
are combined these differences can cancel each
other out (Endsley, 2000). This is consistent
with Landry and Yoo (2012) who recommend
scoring by query rather than a combined score to
avoid the problem of assumed equal likelihood
Evaluation of Objective Measures of SA 9

Table 3: Studies Examining the Sensitivity of SAGAT (and Variants)

Category Number Sensitivity (%)

Domain
 Aviation 17 74
 ATC 17 94
 Driving 32 98
  Health care 19 66
  Power/process control 15 87
 Military 20 90
 Robotics 7 93
  Artificial tasks 6 100
 Other 19 74
Test environment
 Simulation 104 84
 Microworld 26 88
 Video 10 100
  Live exercise 11 77
 Experimental 1 100
Participants
 Experienced 73 84
 Novice 17 59
 General 15 100
 Students 29 88
 Combination 18 100
Method* (p = .02)
 SAGAT 119 89
  End of trial 20 73
 SACRI 4 75
 QUASA 6 67
 SALSA 2 100
 SAVANT 1 50
Analysis* (p = .04)
  By query 43 91
  By SA level 42 92
  Overall score 52 81

Note. SAGAT = Situation Awareness Global Assessment Technique; ATC = air traffic control; SACRI = SA Control
Room Inventory; QUASA = Qualitative Assessment of SA; SALSA = Situation awareness of en-route air traffic
controllers in the context of automation; SAVANT = SA Verification and Analysis Tool; SA = situation awareness.
*p < .05.

of each element. Analysis of SAGAT queries has
also shown a lack of internal consistency (Ends-
ley, 1990b, 2000), arguing against combining
across queries. Nonetheless, many researchers
analyzed SAGAT data as a single score com-
bined across all queries, or as a separate score
for each level of SA. It was found that studies
using a single combined SA score (81% sensi-
tivity) were less likely to be sensitive than those
that were analyzed by query (91% sensitivity) or
by SA level (92% sensitivity; z = 1.77, p = .04).
When just the 68 studies that used SAGAT
as designed (with scenario freezes and analysis
by level or by query) are considered, overall
sensitivity rises to 90%. An additional problem
was found in that some studies collected only 1
or 2 SAGAT freezes across conditions, result-
ing in very small sample sizes, well below the
10 Month XXXX - Human Factors
60 samples per query per condition recom-
mended by Endsley (2000). When the four
studies with very low sample sizes are removed,
overall sensitivity of SAGAT rises to 94%.
It should be noted that the two studies using
the SALSA scoring method performed well
(100%), three out of four SACRI studies were
sensitive (75%), and four of the six studies using
the QUASA method were sensitive (67%).
However, the numbers of studies using these
techniques are so few that it is difficult to con-
clude anything about them.
SPAM and real-time probes. Of the 27
papers that included SPAM or real-time probes
to measure SA, 2 were duplicates and 1 paper
involved 2 studies, making for a total of 26 stud-
ies. These papers are listed in Appendix B,
which is available with the manuscript on the
Human Factors website. The proportion of stud-
ies in which SPAM or a real-time probe was sen-
sitive to the experimental manipulation in the
study is listed in Table 4. The mean sensitivity
for these studies overall is 69.2%.
The majority of the studies were conducted in
aviation, ATC, driving, military, process control,
and submarine management. There were not
sufficient numbers of studies to compare sensi-
tivity across these domains. The majority of the
studies were conducted in simulations, although
a few were also conducted in microworlds,
watching videos, or in live exercises. Partici-
pants were primarily either experienced opera-
tors or students. Neither testing environment nor
participant type significantly affected the sensi-
tivity of the metric (p > .1). SPAM probes (sen-
sitivity = 64%) were actually slightly less sensi-
tive than real-time probes (sensitivity = 73%),
although this difference was not significant (z =
0.53; p = .30).
Limitations. The sensitivity meta-analyses
were based on whether each measure was sensi-
tive in a wide variety of studies, each with dif-
ferent numbers of participants. The use of
p-values for conducting hypothesis testing has
been criticized on the basis that it does not nec-
essarily indicate the magnitude of effects (i.e., it
is subject to the effects of sample size) (Kline,
2004).
However, a p-value threshold for hypothesis
testing has been the standard in the field and is
generally reported across the literature spanning
some 30 years. To examine the possibility that
variations in sample size across studies unduly
affected the sensitivity analyses, an analysis of
variance (ANOVA) was conducted that found
no significant effect of number of subjects on
the sensitivity score of the studies in this meta-
analysis, F(2, 175) = 0.05, p = .95.
Several other caveats also need to be noted
with regard to this meta-analysis of sensitivity.
First, it is certainly possible that a metric could
find a difference between study conditions that
does not exist (i.e., false positives). It is also pos-
sible that the study itself was poorly constructed
with insufficient power, leading to a lack of sen-
sitivity due to study design rather than the metric
itself. While there was no obvious data to sup-
port that either situation was more or less likely
to exist with regard to the studies that used either
SAGAT or SPAM, a further assessment was
made of the studies that employed both mea-
sures within the same study as a means of more
carefully assessing their comparative sensitivity
to SA differences.
Comparison of technique sensitivity. Figure
1 shows the proportion of studies in which each
metric was sensitive to the study manipulation.
When conducted with freezes according to the
prescribed methodology and scored by query,
SAGAT is significantly more sensitive than both
SPAM (z = 14.88, p < .001) and real-time probes
(z = 3.05, p = .001).
Six studies included a direct comparison of
the sensitivity of SAGAT and either SPAM or
real-time queries in the same study. Two studies
found SAGAT more sensitive than real-time
probes (Endsley, Sollenberger, & Stein, 2000;
Jones & Endsley, 2004), three studies found
SPAM more sensitive than SAGAT (Alexander
& Wickens, 2005; Cummings & Guerlain, 2007;
Silva et al., 2017), and one study found mixed
results with SAGAT and real-time probes
equally sensitive to the study manipulations
(Burns et al., 2008).
To further examine these findings, Jones and
Endsley (2004) found that SAGAT was more
sensitive than real-time probes because the sim-
ulation freeze allows more queries to be col-
lected, yielding a more reliable measure of SA
(which is important when periodic sampling is
Evaluation of Objective Measures of SA 11

Table 4: Studies Examining the Sensitivity of SPAM and Real-Time Probes

Category Number Sensitivity (%)

Domain
 Aviation 5 100
 ATC 3 33
 Driving 6 83
  Submarine management 3 33
  Power/process control 3 67
 Military 4 50
 Other 2 100
Test environment
 Simulation 17 65
 Microworld 3 67
 Video 4 100
  Live exercise 2 50
Participants
 Experienced 16 75
 Students 10 60
Method
 SPAM 11 64
  Real-time probes 15 73

Note. SPAM = Situation Present Assessment Technique; ATC = air traffic control.

involved). As SPAM and real-time probes can
only be asked one at a time and since excessive
interruptions are not possible, it is at a disadvan-
tage. Of the three studies where SPAM was more
sensitive than SAGAT, one measured SAGAT
only at the end of two trials, which created insuf-
ficient data (Cummings & Guerlain, 2007). In
Silva et al. (2017), while SAGAT was not sensi-
tive to operator experience level, neither were
the performance measures captured. Experi-
enced operators had better accuracy on SPAM
however.
Predictiveness
A total of 53 comparisons were found across
46 papers that examined the predictiveness of an
objective measure of SA, as shown in Appendix
C, which is available with the manuscript on the
Human Factors website. Overall, 89.6% of the
analyses found SA measures predictive of the
performance or decision-making measures col-
lected in the study.
Table 5 shows that 35 analyses involved
SAGAT (predictiveness = 89%), 4 examined end-
of-trial SA assessment (predictiveness = 100%), 1
involved SALSA (predictiveness = 50%), 1
involved a real-time probe (predictiveness = 0%),
and 12 involved SPAM (predictiveness = 100%).
The sensitivity of SAGAT/SALSA/end-of-trial
assessments compared to SPAM/real-time probes
was not significantly different (z = .37, p = .35).
A total of 43 studies provided correlation data
between the SA measure and performance mea-
sures. In comparing the mean Pearson’s r calcu-
lated for studies using each method, the end-of-
trial measures were most highly predictive of
performance (r = .533), followed by SAGAT
(r = .459), and then SPAM (r = .411). There was
partial overlap in the 95% confidence intervals
that were calculated from the converted Z-scores
for SAGAT and SPAM, however.
Seven papers examined the relative predic-
tiveness of these measurement techniques within
the same study. Two studies found SAGAT to be
more predictive than real-time probes or SPAM
(Jones & Endsley, 2004; Loft, Bowden, et al.,
2015). Two studies found SPAM to be more pre-
dictive than SAGAT (Durso et al., 2006; Krae-
mer & Süß, 2015). The Durso et al. (2006) study,
however, artificially restricted SAGAT to only
12 Month XXXX - Human Factors
Figure 1. Sensitivity of SA Metrics: Probability of detection of SA difference due to
study manipulation.
a
SAGAT with prescribed method and sample size.
one query per freeze, significantly reducing its
sample size unnecessarily. In the Loft, Bowden,
et al. (2015) study, which allowed more queries
during freezes, this handicap disappeared result-
ing in SAGAT accounting for twice the variance
in performance compared to SPAM.
Three studies found mixed results. Durso
et al. (1998) found that SPAM was more sensi-
tive to one performance measure and SAGAT
was more sensitive to another. Pierce, Strybel,
and Vu (2008) similarly found SAGAT and
SPAM to each be more sensitive to different out-
come measures. Strybel et al. (2008) showed
that SPAM latency was a slightly better predic-
tor of pilot airspeed variability than SAGAT
(r2 = .08 vs. r2 = .07), although they used a com-
bined score for SAGAT which is generally less
sensitive. Based on these findings, it would
appear that both SPAM and SAGAT are predic-
tive of performance.
In one counter-intuitive finding, Loft et al.
(2018) found that while SAGAT was predictive
of performance in a submarine management
study, this was only at the between subjects
level, but not at the within subjects level. This
would indicate that the differences in SA
between subjects correlated with performance,
but not changes in an individual’s SA within a
trial, which is hard to explain on any theoretical
level. SAGAT scores have been shown to be
highly stable across individuals (Endsley & Bol-
stad, 1994), so the finding of between subjects
predictiveness is not unexpected.
In examining the lack of a within subjects
effect, the most likely explanation is that in this
study the queries were all referenced to ship con-
tact numbers which creates a significant memory
burden. Previous research has shown that air traf-
fic controllers do not remember aircraft call
signs, for example, but rather rely on spatial
memory of where air traffic is located (Endsley
& Rodgers, 1998). By pinning each query to the
ship contact number in the Loft et al. (2018)
study, rather than a spatial map as is typical with
SAGAT administration, it is possible that the
researchers created an artificial memory require-
ment that interfered with their results. This mat-
ter will require further research. The authors also
indicated that it is possible that different perfor-
mance windows need to be considered to link SA
to performance in their domain.
Intrusiveness
A common concern about both measures is
whether they are intrusive on operator perfor-
mance, SAGAT by freezing the scenario, and
Evaluation of Objective Measures of SA 13
Table 5: Predictiveness of SA Metrics
Method Number Predictiveness (%)
End of trial 4 100
SAGAT 35 89
SPAM 12 100
Real-time probes 1 0
SALSA 1 50
Note. SA = situation awareness; SAGAT = Situation Awareness Global Assessment Technique; SPAM = Situation
Present Assessment Technique; SALSA = Situation awareness of en-route air traffic controllers in the context of
automation.
SPAM by dual tasking the operator while con-
ducting operations. A total of 18 studies were
found that examined the issue of intrusiveness of
one or both of the techniques, shown in Table 6.
Eleven of the papers examined the effect of
SAGAT on operator performance and or work-
load. No negative effect of SAGAT administra-
tion on operator performance was found in any
of these studies. Endsley (1995b) showed that
stopping the simulator to collect SAGAT data
for 30, 60, or 120 s and either 1, 2, or 3 times in
a 20 min scenario had no effect on participant
performance as compared to scenarios in which
there were no SAGAT freezes. In a second study,
I examined whether just the possibility that a
SAGAT freeze would be administered created a
change in performance (Endsley, 2000). In that
study, I told participants there would definitely
not be a freeze in one-third of the trials and that
there might be a freeze in the other two-thirds
(of which half actually experienced a SAGAT
freeze). Again, there was no effect on perfor-
mance from either the SAGAT freeze or the
threat of a SAGAT freeze, showing no “pop
quiz” effect.
Since those studies, an additional nine studies
have compared trials in which SAGAT freezes
were administered to trials without SAGAT
freezes and found no negative effect on perfor-
mance indicating intrusiveness. In one study, a
small increase in workload (2.4%) was reported
in the SAGAT trials (Loft, Bowden, et al., 2015),
but the authors reported that this may have been
due to the fact that twice as many queries were
provided in the SAGAT trials than in SPAM tri-
als. In another study (Kraemer & Süß, 2015), an
(95% Confidence
Mean Pearson’s r Interval)
.533 [.522, .545]
.459 [.432, .487]
.411 [.368, .454]
— —
.184 —
improvement in performance was shown in the
trial that included SAGAT; however, this effect
was confounded with trial order and likely
showed a learning effect for the very inexperi-
enced participants in that study.
In contrast, 6 of 15 studies (40%) that used
SPAM or real-time probes reported a negative
effect on operator performance or workload.
Pierce (2012) found a negative affect of SPAM
on both operator performance and workload.
Loft et al. (2016) also found that experts in their
study (submarine operators) took almost 20 s to
accept SPAM queries, presumably due to work-
load issues, calling into question its validity.
Reportedly subjects delayed answering the
probes when SA was lower and uncertainty
higher. Similarly Shelton, Kinston, Molyneux,
and Ambrose (2013) reported that their partici-
pants (physicians) found the verbal probes intru-
sive and interfered with team dialog. Some que-
ries were not answered at all, presumably due to
workload issues. Four studies reported a nega-
tive effect of SPAM on concurrent performance,
as compared to trials in which SPAM was not
administered (Pierce, 2012; Pierce, Strybel, &
Vu, 2008; Pierce, Vu, et al., 2008; Strybel et al.,
2008).
Based on these studies, SPAM appears to cre-
ate significant interference with performance
and increases operator workload. Keeler et al.
(2015) implemented SPAM with computerized
queries rather than verbal queries, similarly to
Bacon and Strybel (2013) and Silva et al. (2013).
None of these 3 studies found SPAM to be intru-
sive. However, their instructions were to only
answer the SPAM queries if the operator felt
 Table 6: Intrusiveness of SA Metrics

14
Study Domain Environment Subjects SAGAT SPAM Notes

Durso et al. (1998) ATC Microworld Students No effect on No effect on performance Only 1 SAGAT question/stop
performance
Durso, Bleckley, and ATC Microworld Students No effect on No effect on performance or  
Dattel (2006) performance or workload
workload
Bacon and Strybel (2013) ATC Simulation Novice — No effect on performance or Trend toward faster RT on conflicts of
workload probed aircraft
Endsley (1995b) Aviation Simulation Experienced No effect on — 0, 1, 2, or 3 stops of 30, 60, or 120 s
performance
Endsley (2000) Aviation Simulation Experienced No effect on — Performance not effected by SAGAT
performance stop or by possibility of SAGAT stop
Hogg, Folles, Strand-Volden, Nuclear Simulation Experienced No effect on — (SACRI) Subjective performance
and Torralba (1995) Power performance
Keeler et al. (2015) ATC Simulation Novice No effect on performance Computer version of SPAM
Kraemer and Süß (2015) ATC Simulation Students Positive effect on No effect on performance Confounded with order effect (i.e.,
performance improvement due to learning)
Loft, Bowden, et al. (2015) Submarine Microworld Students No effect on No effect on performance or More queries with SAGAT (2%
performance, small workload increase in workload)
effect on workload
Loft et al. (2016) Submarine Microworld Experienced and — Negative effect on performance Experts took almost 20 s to accept
students SPAM queries—longer when SA
lower and uncertainty higher
C. A. Morgan, Chiappe, Kraut, ATC Microworld Novices No effect on No effect on workload  
Strybel, and Vu (2012) workload
Pierce (2012) ATC Microworld Students — Negative effect on performance  
& workload
Pierce, Vu, Nguyen, and ATC Microworld Students — Negative effect on performance  
Strybel (2008)
Pierce, Strybel, and Vu (2008) ATC Microworld Students No effect on Negative effect on performance SAGAT end of trial
performance
Shelton, Kinston, Molyneux, Health Care Simulation Novice — Negative effect on discussion Verbal queries viewed as intrusive,
and Ambrose (2013) confused with dialog
Silva et al. (2013) ATC Simulation Novice — No effect on performance Computer version of SPAM
Snow and Reising (1999) Aviation Simulation Experienced No effect on No effect on performance  
performance
Strybel, Vu, Kraft, and Aviation Simulation Experienced No effect on Negative effect on performance Performance worse on SPAM trials
Minakata (2008) performance compared to SAGAT trials

Note. SAGAT = Situation Awareness Global Assessment Technique; SPAM = Situation Present Assessment Technique; ATC = air traffic control; RT = response time; SACRI = SA
Control Room Inventory; SA = situation awareness.
Evaluation of Objective Measures of SA 15
able. This results in a skewing of SA probes into
low workload periods, as will be discussed in a
later section.
These results indicate that concerns over the
potential intrusiveness of SAGAT are unwar-
ranted, with no negative effects on performance
in any of the studies that examined it. It should
be noted that the reason SAGAT is not intrusive
on performance is that during the freeze period
operators are actively working with their situa-
tion knowledge to answer the queries (Endsley,
1995b). This keeps the situational information
active in memory.
This is a very different situation than inter-
ruptions in which a person conducts unrelated
tasks or answers unrelated questions, which has
been shown to be detrimental to both SA and
performance (Gartenberg, Breslow, McCurry, &
Trafton, 2014; Loft, Sadler, et al., 2015;
McGowan & Banbury, 2004; Ratwani & Traf-
ton, 2008). Negative effects can also be incurred
with other types of interference that are not con-
sistent with the SAGAT methodology. For
example, Tremblay, Vachon, Lafond, & Kramer
(2012) required people to recover SA of changed
situations by failing to pause the simulation
while displays were blanked, negatively effect-
ing task performance. McGowan and Banbury
(2004) provided 10-s interruptions in 20- to
25-s-long video clips that were passively
viewed, showing both that irrelevant questions
negatively affected performance and that rele-
vant questions improved performance (even
when the trial was not interrupted). This very
short, highly artificial task is not representative
of the normal types of domain relevant and
engaging simulations that SA data are collected
in. It likely became a task of memorization for
the very narrow set of items that were ques-
tioned during the study. In contrast, SAGAT is
generally only collected two to three times, no
closer than 3 min apart in a 20-min interactive
scenario where participants are engaged in nor-
mal task performance, and provides a wide
range of queries that cannot be effectively pre-
pared for in advance.
Memory Dependence
Another concern about SAGAT has been
whether it is overly reliant on working memory.
This discussion rests partially on whether people
in complex domains need to gather and integrate
information in memory to make relevant, timely
decisions, or whether one believes that simply
looking up information as needed is sufficient
for expert performance. In time-critical domains
like driving, medicine, aviation, and ATC, it is
clear that the former is the case.
Endsley (2015b) provides a detailed discus-
sion on the role of memory in SA, showing that
inexperienced operators’ SA is often constrained
by working memory limitations, but that experts
have access to long-term memory stores that
largely circumvent these limits. The question for
SA measurement is whether operators can report
SA data during freezes, or whether their knowl-
edge will rapidly decay over the period of the
freeze. Working memory generally lasts about
20 to 30 s unless it is kept activated (Baddeley,
1986).
To investigate this question, Endsley (1990a)
provided pilots with SAGAT queries in random
order during freezes. This study showed that
pilots were equally accurate in answering
SAGAT queries across a 6-min period after a
freeze, indicating no memory decay. Endsley
(1990a, 2000) concluded that this result supports
a model of cognition in which working memory
is an activated subset of long-term memory
(Cowan, 1988). This model is consistent with
many others, including Durso and Gronlund
(1999), Adams, Tenney, and Pew (1995), and
Sarter and Woods (1991).
A number of studies were found in this review
that refute the charge that SAGAT is overly
dependent on working memory, shown in
Appendix D, which is available with the manu-
script on the Human Factors website. Endsley
and Bolstad (1994) showed that primary work-
ing memory ability did not predict SAGAT
scores in experienced pilots. Sulistyawati, Wick-
ens, and Chui (2011) also found that experienced
pilots answered SAGAT questions equally well,
irrespective of their scores on memory span
tests. Jipp and Ackerman (2016), similarly,
found no relationship between primary working
memory scores and SAGAT in an ATC task with
student participants.
Gonzalez and Wimisberg (2007) showed that
Level 1 SA, as measured by SAGAT, improved
16 Month XXXX - Human Factors
over time with experience on a task, and its rela-
tionship to a visual working-memory measure
decreased accordingly. Gutzwiller and Clegg
(2012) found no relationship between working
memory and Level 1 SA in their study, based on
an end-of-trial SAGAT measure; however, it
was correlated with Level 3 SA which would be
required for complex reasoning and projection
per the Endsley (1995c) model of SA.
Two studies show somewhat different con-
clusions. Durso et al. (2006) reported that
SAGAT scores correlated with measures of
complex working memory (Operational Span
and Reading Span) as well as fluid intelligence,
but not measures of primary working memory or
visual memory. It should be noted that they used
inexperienced subjects from the general popula-
tion performing in an ATC simulation. They also
provided questions about the situation that were
not necessarily reflective of the SA needed to
perform the task, such as “at what level will the
next airplane to have a heading change be?”
This type of question requires extensive manip-
ulation of information in working memory to
answer, as compared to typical SAGAT queries
that ask for information that should already be
known to perform the task (e.g., what is the next
sector of the indicated aircraft?).
Cak, Say, and Misirlisoy (2019) found that
both SAGAT and SPAM were correlated with an
Operational Span complex working memory
test in a study with experienced pilots. (In addi-
tion, both measures were correlated with the
number of hours the pilots had spent in the simu-
lator and SPAM was also correlated with a mea-
sure of divided attention.) The fact that SPAM as
well as SAGAT correlated with this complex
working memory measure is puzzling in that all
information is visually present when answering
SPAM probes (although since SPAM includes
queries that focus on the past, it introduces its
own memory component).
Cak, Say, and Misirlisoy explained that their
finding runs counter to previous studies due to
the different measure of working memory
employed in their study—the Operation Span
test. This test requires people to actively main-
tain information in memory while performing
secondary processing tasks and has been shown
to be significantly correlated with the ability to
control attention in numerous studies (Kane,
Bleckley, Conway, & Engle, 2001; Poole &
Kane, 2009; Unsworth & Engle, 2005; Unsworth
& Spillers, 2010).
Primary memory (the ability to hold informa-
tion in memory), attentional control, and the
ability to recall information from secondary
memory have been shown to be separate aspects
of cognition that all contribute to Operation
Span—a measure of complex working memory
(Shipstead, Lindsey, Marshall, & Engle, 2014).
As concluded by Shipstead et al (2014)
these mechanisms are not similarly rep-
resented by all working memory tasks.
Running memory span performance
reflects primary memory more strongly
than either complex span or visual arrays
tasks. The performance of these later tasks
is more closely associated with a persons’
attentional control and retrieval abilities.
(p. 138)
Therefore, it appears that both Durso et al.
(2006) and Cak et al. (2019) employed a com-
plex span working memory measure that is more
closely representative of factors such as atten-
tional control and secondary memory aspects,
whereas the other studies that examined the role
of working memory on SA have operationalized
it via tests which more closely reflect primary
memory.
From the standpoint of SA, therefore, it is not
too surprising that both SAGAT and SPAM per-
formance correlated with the Operation Span
test, reflecting the importance of attentional con-
trol on SA. What is surprising is that Cak et al.
(2019) only found SPAM to be correlated with a
measure of attention sharing ability, whereas
Endsley and Bolstad (1994) found attention
sharing to be highly predictive of SAGAT in
pilots. From the standpoint of the present ques-
tion, it does not appear that SAGAT is overly
reliant on the need for people to hold informa-
tion in primary (working) memory with experi-
enced participants; however both SAGAT and
SPAM do have correlations with attentional
measures, reflecting its importance to SA.
Therefore, with experienced participants col-
lection of SAGAT data is not working memory
constrained when the information is collected
Evaluation of Objective Measures of SA 17

immediately following a freeze, for at least 2 to answers. In a meta-analysis of 65 studies, Vidu-

3 min afterwards, as they have access to relevant lich (2000) found that probe measures are most
information in longer term memory representa- sensitive when a wide range of queries are pro-
tions. Novices and inexperienced participants vided, rather than just a few. This is likely
are likely more reliant on working memory for because it precludes trying to memorize answers
processing information and typically have lower in advance of a freeze. The unpredictability of
SAGAT scores than experienced participants when the freezes will occur also avoids memori-
(Hogan, Pace, Hapgood, & Boone, 2006; Jack- zation and effects on performance (Endsley,
son, Chapman, & Crundall, 2009; Kass, Cole, & 1995b, 2000).
Stanny, 2007; Randel, Pugh, & Reed, 1996;
Soliman & Mathna, 2009; Strater, Endsley, Ple- Speed-Accuracy Tradeoffs
ban, & Matthews, 2001). Even so, it is interest-
A number of studies found that answers to
ing to find that there were not significant differ-
SPAM probes were often more accurate than
ences in the sensitivity of SAGAT based on the
similarly worded SAGAT queries (Durso et al.,
experience level of the participants in the 152
2006; Pierce, Strybel, & Vu, 2008; Strybel
studies reviewed here (p >.05). SAGAT was sen-
et al., 2008), which is not surprising considering
sitive with both experienced and inexperienced
that the correct answer is clearly visible when
participants. Overall these findings are consis-
responding to a SPAM probe.
tent with recent work showing that with increas-
While SPAM reaction time theoretically
ing levels of domain expertise people rely more
measures how cognitively available information
on running memory, actively updating and pro-
is, a number of studies reviewed show that it
cessing new information as the task proceeds,
may instead be measuring look up time. SPAM
as opposed to passive working memory that
and SAGAT differ in two main respects: SAGAT
simply tries to memorize information without
pauses the simulation and hides displays during
further processing (Anderson-Montoya, Scerbo,
query administration, while SPAM leaves them
Ramirez, & Hubbard, 2017).
visible and does not pause the simulation. C. A.
It should also be noted that a few studies
Morgan, Chiappe, Kraut, Strybel, and Vu (2012)
using SAGAT applied it to very short, artificial
performed a study comparing the same ques-
tasks that may not be suitable. Gugerty (1998),
tions provided when the display was visible ver-
for example, provided very short scenarios (18–
sus not visible and when the simulation was
35 s) and then assessed driver’s ability to recall
paused or not paused. They found that SA que-
the location of other vehicles. deWinter et al.
ries were more accurate when the displays
(2018) provided participants with a set of six
remained visible, as would be expected, but that
gages to monitor with instructions to report
response time was almost 2 s slower, revealing a
when a gage exceeded a limit. Even though
significant speed-accuracy tradeoff. Several
SAGAT was sensitive in these studies, they are
other studies have also found speed-accuracy
not realistic examples of real-world tasks. Such
tradeoffs for SPAM and real-time probe mea-
artificial tasks likely leave participants with little
sures (Alexander & Wickens, 2004, 2005; Jones
to do but attempt to memorize information.
& Endsley, 2004; Taber, McCabe, Klein, &
In contrast, SAGAT was designed for realis-
Pelot, 2013). In general, it appears that people
tic, ecologically valid simulations, and it is rec-
will look to check before answering queries if
ommended that no SAGAT freezes occur for at
the information is available, showing that SPAM
least the first 3 min (and with at least 3 min in
more accurately assesses the time for people to
between freezes) in order that participants
look-up information on a display, regardless of
become engaged in performing their tasks (End-
whether they know the answer.
sley, 1995b, 2000). Participants are instructed to
perform their tasks as usual and to simply answer
the SAGAT questions to the best of their ability. Sampling Bias
Because SAGAT queries are representative An additional concern is whether the meth-
across a wide sample of normal SA require- ods contain any sampling bias. Because SAGAT
ments, this precludes trying to memorize recommends conducting the freezes at random
18 Month XXXX - Human Factors
times, the likelihood of sampling bias (e.g., by
collecting during only high workload or low
periods, or just during interesting events) is
minimized. While Lau, Jamieson, and Skraan-
ing (2016) advocate for collecting SA at only
important times as indicated by a subject matter
expert, this can cue subjects and provide arti-
ficial information affecting their performance
(Endsley, 1995b, 2000).
SPAM provides probes at random intervals as
well; however, participants are able to delay
answering, or may even neglect to answer.
Durso et al. (1998) noted that participants could
take as long as 10 s to answer the call line to get
a SPAM probe in their study, and as long as 4 s
to answer a query. Loft et al. (2016) found their
participants took as long as 20 s to accept que-
ries, taking longer when SA was low or uncer-
tainty was high. Trapsilawati, Wickens, Qu, and
Chen (2016) reported means latencies in accept-
ing a SPAM probe of between 9 and 23 s. Stry-
bel, Vu, Battiste, and Johnson (2013) reported
that 5% of probes went unanswered, and Cun-
ningham et al. (2015) found that pilots did not
respond the ready prompt 16% of the time in
their study due to workload. Further two addi-
tional studies showed that SPAM latency mea-
sures are correlated with NASA TLX scores,
indicating that it is associated with workload
(Strybel et al., 2010; Strybel et al., 2008).
These results indicate that participants may
delay attending to SPAM probes until lower
workload periods. While the time to accept a
probe is a secondary workload indicator, this
ability to delay answering probes, or to ignore
them completely, creates a bias in the measure in
favor of lower workload periods. While Bacon
and Strybel (2013), Silva et al. (2013), and Kee-
ler et al., (2015) reported that instructing partici-
pants to only answer SPAM queries if they felt
able to and providing computerized SPAM que-
ries rather than verbal queries could reduce
SPAM intrusiveness, this method also system-
atically skews SA probes into low-workload
periods.
SA vs. Workload
Another theoretical concern is the extent to
which SPAM is measuring SA or workload.
While high workload can lead to low SA, it has
also been shown that SA and workload can be
independent across much of the demand scale,
with low SA possible in low-workload periods
(vigilance) and both high and low SA pos-
sible across moderate workload levels (Endsley,
1993a; Vidulich & Tsang, 2015). It is therefore
important that SA measures be clearly distin-
guishable from workload measures.
The fact that operators must answer real-time
probes while conducting operational tasks cre-
ates a situation in which it serves as a secondary
workload measure. Jones and Endsley (2004)
found a weak correlation between real-time
probes and a concurrent measure of workload.
This issue is somewhat reduced with the intro-
duction of a ready prompt with the SPAM tech-
nique, which allows participants to answer the
probes while they have more spare capacity, but
it may not be completely eliminated.
Pierce, Strybel, and Vu (2008), for example,
found that SPAM latency measures were not
predictive when participants were under high
workload because they neglected the SPAM
probes while meeting task demands. Pierce,
Strybel, and Vu (2008) also found correlations
between workload subscale ratings on trials that
used SPAM and those that involved other types
of secondary tasks. Given that Pierce (2012)
found a negative effect of SPAM on workload,
and that 40% of the studies reported on here that
used SPAM found negative effects on perfor-
mance, it seems likely that SPAM is providing
secondary task loading. Coupled with the find-
ing that people frequently look up the answer to
SPAM probes, which is a secondary task, there
is considerable evidence that SPAM likely mea-
sures workload to some degree, rather than SA.
Query Design
Whichever measure is used, the actual que-
ries provided are critical to obtaining valid
research results. Variability between the stud-
ies reviewed in this study may be due to (a)
whether they employed a systematic analysis
of SA requirements for the domain under study,
(b) construction of clear queries that can be
consistently scored, (c) differences between
SAGAT and SPAM in the types of information
solicited, and (d) methods for soliciting partici-
pant responses.
Evaluation of Objective Measures of SA 19
Reflection of SA requirements.  One observed
issue with SPAM, and with some researchers
use of SAGAT, has been the development of
queries that may in fact have little to do with the
operator’s SA requirements. For example, Durso
et al. (1998) posed “Which has the lower alti-
tude, TWA799 or AAL957?” While that probe
undoubtedly requires knowledge of situational
information to answer (either from memory or
from displays), it is not something that air traffic
controllers typically need to know to do their
jobs (Endsley & Rodgers, 1994b). Similarly,
Chiappe et al. (2015) report providing queries
such as “are the majority of the aircraft in your
sector westbound?” Someone looking at a dis-
play could answer that question, but it is quite
extraneous to what is required for good SA in
ATC (Endsley & Jones, 1995; Endsley & Rodg-
ers, 1994a). Asking questions about the situation
is not the same as asking questions that are per-
tinent to SA. Whichever technique is used, a
detailed analysis of SA requirements for the
operational role being examined is extremely
important (Endsley, 1995b). While many of the
studies reviewed reported conducting a goal-
directed task analysis (GDTA) or other analysis
of the task domain to determine their questions,
others did not.
Query construction. Second, crafting clear,
concise questions that can be objectively scored
as correct or incorrect is important. The ques-
tions should not be subjective or ambiguous, and
should produce the same answer from experts
with full knowledge of the situation. For exam-
ple, Lau, Jamieson, and Skraaning (2014) had
trouble with ill-defined queries that could not be
consistently scored by three experts who were
actually looking at the same displays, even with
all the data visible and all the time needed,
resulting in low inter-rater reliability in evaluat-
ing their queries. While typically the query scor-
ing key is determined via data in the simulation
computer, this study points to the importance of
creating clear and relevant queries.
Query focus. A notable difference between
SPAM and SAGAT is that SPAM creates ques-
tions that test a participant’s knowledge of the
past, present, and the future, while SAGAT
questions are directed at the perception, compre-
hension, and projection components of SA.
SPAM-future and SAGAT-projection queries
seem clearly aligned. SPAM-present and
SAGAT-perception also seem to be aligned,
although SPAM-present may also potentially
include SAGAT-comprehension questions.
SPAM includes questions on the past under
the belief that remembering past events is impor-
tant for projecting future events (Durso et al.,
2006). This, however, adds a clear memory com-
ponent to SPAM that may or may not be relevant
to current operations. SA is generally considered
to be a person’s knowledge and understanding of
the present, and its reliance on what has occurred
in the past is only relevant to the extent that it
affects the present or future, for example, under-
standing the trend in some variable (Endsley,
1995c). SAGAT may directly ask whether some
relevant variable (e.g., altitude or speed) is
increasing, decreasing, or staying level, for
example, but would not ask what it was at some
prior point in time. In other words, while SPAM
asks specifically for memory of past events or
status, SAGAT asks for how information is
changing or how it will change in the future, not
specifically requiring reporting on memory of
the past. Thus, SPAM may be more reflective of
memory than SAGAT in this respect.
Query presentation format.  A final issue has
to do with the referents for queries that are pro-
vided. It is recommended that SAGAT queries
start with a domain relevant map that allows par-
ticipants to input the relevant objects (e.g., air-
craft, automobiles, friendly, and enemy forces)
that they know about in an intuitive spatial lay-
out (Endsley, 1995b, 2000). Subsequent queries
refer back to the objects entered on that map.
This is because identifiers (e.g., aircraft call
signs, track numbers, train numbers) are often
not known, but rather are peripheral information
that serve a functional task, but have little value
otherwise in terms of situation understanding
(Endsley, 2015a; Endsley & Rodgers, 1998).
Implementations of SAGAT measures that rely
heavily on such identifiers to answer queries
create an artificial memory requirement that is
likely to lead to difficulties such as experienced
by Golightly, Wilson, Lowe, and Sharples
(2010) in train operations, or Loft et al. (2018) in
submarine management. Also Ratwani and Traf-
ton (2008) found that spatial memory (such as
20 Month XXXX - Human Factors
that provided with maps in SAGAT) is impor-
tant for resumption from an interruption.
Conversely, SPAM is heavily reliant on iden-
tifiers, but this does not cause a problem because
they are readily available on displays. And Stry-
bel et al. (2016) found that participants per-
formed more poorly on SPAM with graphically
presented probes as compared to verbal probes,
most likely due to physical interference with the
ongoing task, or potentially because they were
presented in a way that made guessing less easy.
In contrast, Shelton et al. (2013) created a text
based display for SPAM probes on a small hand-
held computer because verbally presented que-
ries interfered with normal discussions in a study
with health care teams.
Team SA
In addition to assessing individual SA, many
researchers are also interested in the SA of
teams. A total of 27 studies were found that used
either SAGAT or SPAM to assess SA at the team
level, shown in Appendix F, which is available
with the manuscript on the Human Factors
website. Because SPAM does not elicit data
from multiple operators at the same time, and
queries and responses are generally provided
verbally, an assessment of shared or team SA is
logistically challenging with the technique, and
only two papers were found that used SPAM for
assessing team SA.
A key advantage of SAGAT is that it allows
for the assessment of team SA (Bolstad & Ends-
ley, 2003) because it is administered to all par-
ticipants in a simulation at the same time. Prob-
lems with tools and bottlenecks in team pro-
cesses where information is not passed, or where
different interpretations are made from data that
could lead to poor team performance can be
readily identified by comparing the SA of differ-
ent team members or sub-teams at the same
point in time. A total of 24 studies were found
that used SAGAT to evaluate team SA, 1 used
SACRI, and 1 used QUASA.
These measures were used to assess team SA
in several ways including
1. Creating a combined or average SA score
across the team as an overall Team SA score
(11 studies);
2. Allowing for a collaborative team response to
queries (two studies);
3. Assessing Shared SA (or SA similarity) as the
degree of concurrence of teammates on infor-
mation elements that are relevant to both roles
(14 studies);
4. Examining the degree to which team members
are aware of the SA of each other (Team Meta-
SA) (three studies);
5. Assessing the correlation between the SA of
different team members, or sub-teams (six
studies).
The SA of teams was assessed in a variety of
domains including military (10), health care (5),
aviation (4), emergency management (4), pro-
cess control (3), and business (1) settings. While
most studies were conducted in simulators or
microworlds, two studies were conducted in live
exercises.
Although the exact methods for assessing
SA in teams varied among these studies, a few
general statements can be made. Shared SA as
measured by SAGAT was shown to be predic-
tive of overall team performance (Bonney,
Davis-Sramek, & Cadotte, 2016; Coolen, Draa-
isma, & Loeffen, 2019; Rosenman et al., 2018),
as well as when it was measured by SPAM
(Cooke, Kiekel, & Helm, 2001). Combining
SA scores into an overall Team SA score gener-
ated mixed results, with five studies showing it
to be predictive of overall team performance
(Cooke et al., 2001; Crozier et al., 2015; Gard-
ner, Kosemund, & Martinez, 2017; Parush
et al., 2017; Prince, Ellis, Brannick, & Salas,
2007), and three that did not (Brooks, Switzer,
& Gugerty, 2003; P. Morgan et al., 2015;
Sorensen, Stanton, & Banks, 2010). This body
of research also demonstrated the effect of
team membership on the SA of individuals
(Bolstad & Endsley, 2003; Cuevas & Bolstad,
2010; Leggatt, 2004; Sætrevik, 2012; Seet,
Teh, Soo, & Teo, 2004), as well as the signifi-
cant effect of the team leader (Cuevas & Bols-
tad, 2010), although one study did not find a
significant effect of team membership on
Shared SA (Sætrevik, 2012).
In addition, measures of combined Team SA
and Shared SA were shown to be sensitive to
other aspects of teams, including team knowl-
edge (Bolstad, Cuevas, Gonzalez, & Schneider,
Evaluation of Objective Measures of SA 21
2005; Cooke, Stout, Rivera, & Salas, 1998),
team cohesion and coordination (Hallbert,
1997), team experience (Crozier et al., 2015),
organizational hub distance (Bolstad et al., 2005;
Saner, Bolstad, Gonzalez, & Cuevas, 2009),
team displays (Javed, Norris, & Johnston, 2012),
information flows (Artman, 1999), and stress
(Price & LaFiandra, 2017).
Only a few studies have examined team meta-
awareness, with Sulistyawati, Chui, and Wick-
ens (2008) and Sulistyawati et al. (2009) show-
ing that fighter pilots’ SA was moderately cor-
related with their teammates, but awareness of
the teammates’ SA was not predictive of perfor-
mance. Yuan, She, Li, Zhang, and Wu (2016)
found that awareness of teammate SA was nega-
tively correlated with own SA, however, likely
due to competing task demands.
While the SA of teams has often been investi-
gated by examining team communications or
coordination metrics, or by an examination of
the types of processes they engage in, this work
shows that objective measurement of the overall
Team SA or Shared SA that people have when
working in teams can provide useful comple-
mentary information to this body of research.
Conclusions
In conclusion, this analysis examined 243
papers that measured SA using SAGAT, SPAM,
or one of their variants, and provided a useful
examination of concerns regarding these tech-
niques. While both SAGAT and SPAM were
found to be equally predictive of performance,
this analysis shows that SPAM has significantly
lower sensitivity than SAGAT (64% compared
to 94%) and is far more intrusive, with 40%
of studies showing an effect of SPAM on task
workload or performance. In addition, issues
with speed-accuracy trade-offs, correlations
with workload, and a sampling bias in favor
of lower workload periods create considerable
concern over SPAM’s validity as a measure of
SA. Evidence of any benefit of “situatedness”
during the SA probe is lacking.
When administered during simulation freezes
and with adequate sample size, SAGAT was
shown to have a very high level of sensitivity to
study manipulations at 94%. Sensitivity was
shown to be much higher when the analysis was
based on accuracy of individual queries or SA
levels, rather than an overall combined SA score.
Studies that collected only a few samples or that
collected data only at the end of the trial had
lower sensitivity.
This analysis shows that claims that SAGAT
is intrusive (Chiappe et al., 2012; de Winter
et al., 2019; Durso et al., 1998; Salmon et al.,
2011; Sarter & Woods, 1991) are unwarranted,
with no negative effects on performance found
in the 11 studies that examined it. The conditions
of SAGAT administration are significantly dif-
ferent than interruptions with unrelated tasks.
This analysis also lays to rest many other poten-
tial criticisms of SAGAT. Concerns about the
over-reliance of SAGAT on working memory
(de Winter et al., 2019; Gugerty, 1998; Jeannot,
Kelly, & Thompson, 2003; Langan-Fox, San-
key, & Canty, 2009; Stanton, Salmon, Walker, &
Jenkins, 2010) were found to be unsupported.
Most SA measurement examined in the pres-
ent analysis was conducted in medium to high
fidelity simulations, or in computer micro-
worlds, both to good effect. While the ability to
administer SA queries in real-world situations is
often cited as a potential benefit of SPAM as
compared to SAGAT, it was interesting to find
that SPAM was only used in two live exercises,
while SAGAT was used in nine. All of these
were controlled studies, rather than actual real
world activities.
SAGAT also was found to provide valuable
insights into the nature of Team SA and Shared
SA in a number of studies. The ability to deter-
mine the outcome of team communications,
coordination processes, or compositions pro-
vides a useful addition to studies in this area.
This is in contrast to claims that SAGAT is not
suited to study of team SA (Langan-Fox et al.,
2009; Stanton et al., 2017).
A few other spurious criticisms of SAGAT
have been made that also deserve comment.
Claims that SAGAT does not capture situation
dynamics (Langan-Fox et al., 2009; Salmon
et al., 2008) are perplexing in that many queries
do indeed cover issues of how key information
is changing (e.g., whether aircraft are ascending
or descending in Table 1), as well as how infor-
mation is projected to change over time as repre-
sented in Level 3 SA questions.
22 Month XXXX - Human Factors
Additional concerns have been raised over
whether SA queries, which tap into operator’s
explicit awareness, are in some way unreflective
of implicit awareness (de Winter et al., 2019;
Gugerty, 1997; Lo, Sehic, Brookhuis, & Meijer,
2016). Endsley (1995c) describes ways in which
automatized performance may be accomplished
without much conscious awareness (e.g., a
driver following a well known route). While this
allows for effective performance in some cir-
cumstances, the resultant lower conscious
awareness can be a problem if something novel
occurs (e.g., the driver would miss a new stop
sign, or a police officer looking for speeders). I
argue that this represents one of the ways in
which SA (as an explicit measure of situation
knowledge) can sometimes diverge from perfor-
mance outcomes. Despite this possibility,
SAGAT was shown to be highly predictive of
performance.
Other domain specific criticisms of SAGAT
are also largely unsupported. Langan-Fox et al.
(2009) and Jeannot et al. (2003), for example,
claim that SAGAT for ATC (a) considers all air-
craft equal, which is not the case (for example,
see Endsley, Mogford, & Stein, 1997; Endsley,
Sollenberger, & Stein, 1999), (b) requires expen-
sive simulators (see also Salmon, Stanton,
Walker, & Green, 2006), even though many
studies use inexpensive microworlds or com-
puter games, and (c) is not suited to multi-sector
studies, which is not accurate as many studies
involve multiple sectors (e.g., Endsley & Rodg-
ers, 1998). Overall SAGAT had 94% sensitivity
in ATC studies.
SAGAT was successfully used in 20 military
studies, including 5 in distributed team command
and control, despite claims that it is unsuitable
for this domain (Salmon et al., 2006; Stanton
et al., 2010). Similarly, Golightly et al. (2010)
concluded SAGAT was not suited for SA mea-
surement in the train driving domain, based on a
short study in which they only collected a small
amount of data at the end of the trial, signifi-
cantly reducing its sensitivity. Rose et al. (2019)
found it sensitive in train driving, however.
As a caveat, the present meta-analysis is con-
strained by the available data and statistics
reported on in the various studies reviewed.
Some have criticized the use of hypothesis
testing reliant on p-values on the basis that such
statistics are often misinterpreted, do not indi-
cate the magnitude of effects, and that statistical
assumptions are rarely tested, instead favoring
the reporting of confidence intervals and effects
sizes (Cumming, 2014; Kline, 2004). However,
others have pointed out that these statistical
approaches are reliant on the same statistics as
p-values and that many concerns are over-
generalized (Kennedy, 2015; van der Linden &
Chryst, 2015). In the present research, a consis-
tent p-value of .05 was used in comparing stud-
ies, as well as a wider p-value of .10 to consider
trend data, although very few studies were added
with this more lenient criterion.
In addition, variance in research results may
have been introduced by the manner in which
different researchers have employed the SA
methods, which was specifically examined in
the analyses to the extent possible and reported
on. Finally, variance in research results may be
due to factors associated with study designs. The
analysis considered the effects of domain, exper-
imental condition, participant type, and number
of subjects, which were largely insignificant;
however, there could be other unknown sources
of variances between the studies.
In conclusion, many of the methodological
concerns regarding objective measurement of
SA via SAGAT can be laid to rest based on this
meta-analysis of research on SA over the past 30
years. When implemented as designed, SAGAT
can provide valuable insights into the effects of
display designs, training implementations, auto-
mation effects, individual differences, and
expertise, and into the construct of SA in both
individuals and teams.
Key Points
•• A meta-analysis of studies employing direct,
objective measures of SA found both SAGAT and
SPAM to be predictive of performance.
•• SPAM was found to be less sensitive than SAGAT,
intrusive on primary task performance and to suf-
fer from speed-accuracy trade-offs, correlations
with workload, and a sampling bias in favor of
lower workload periods, which make it problem-
atic for the measurement of SA.
•• SAGAT was found to have a high level of sen-
sitivity (94%) when employed using the freeze
Evaluation of Objective Measures of SA 23
technique and analyzed by query or SA level, not
to suffer from primary memory constraints, and
not to be intrusive on task performance across a
large number of studies. It provides useful insights
into both individual and team SA in a wide variety
of domains and experimental settings.
ORCID iD
Mica R. Endsley https://orcid.org/0000-
0002-2359-947X
Supplemental Material
The online supplemental material is available
with the manuscript on the Human Factors website.
References
Adams, M. J., Tenney, Y. J., & Pew, R. W. (1995). Situation aware-
ness and the cognitive management of complex systems.
Human Factors, 37, 85–104.
Alexander, A. L., & Wickens, C. D. (2004). Measuring traffic
awareness in an integrated hazard display. Proceedings of the
Human Factors and Ergonomics Society Annual Meeting, 48,
171–175.
Alexander, A. L., & Wickens, C. D. (2005, April 18–21). Synthetic
vision systems: Flightpath tracking, situation awareness, and
visual scanning in an integrated hazard display. Proceedings
of the International Symposium on Aviation Psychology, Okla-
homa City, OK.
Anderson-Montoya, B. L., Scerbo, M. W., Ramirez, D. E., & Hub-
bard, T. W. (2017). Running memory for clinical handoffs: A
look at active and passive processing. Human Factors, 59,
393–406.
Artman, H. (1999). Situation awareness and co-operation within
and between hierarchical units in dynamic decision making.
Ergonomics, 42, 1404–1417.
Bacon, L. P., & Strybel, T. Z. (2013). Assessment of the valid-
ity and intrusiveness of online-probe questions for situation
awareness in a simulated air-traffic-management task with stu-
dent air-traffic controllers. Safety Science, 56, 89–95.
Baddeley, A. D. (1986). Human memory. Oxford, UK: Clarendon
Press.
Bolstad, C. A., Cuevas, H., Gonzalez, C., & Schneider, M. (2005,
May 16–19). Modeling shared situation awareness. Paper pre-
sented at the 14th Conference on Behavior Representation in
Modeling and Simulation (BRIMS), Los Angeles, CA.
Bolstad, C. A., & Endsley, M. R. (2003). Measuring shared and
team situation awareness in the army’s future objective force.
Proceedings of the Human Factors and Ergonomics Society
Annual Meeting, 47, 364–373.
Bolstad, C. A., Foltz, P., Franzke, M., Cuevas, H. M., Rosenstein,
M., & Costello, A. M. (2007). Predicting situation awareness
from team communications. Proceedings of the Human Fac-
tors and Ergonomics Society Annual Meeting, 51, 789–793.
Bonney, L., Davis-Sramek, B., & Cadotte, E. R. (2016). “Think-
ing” about business markets: A cognitive assessment of market
awareness. Journal of Business Research, 69, 2641–2648.
Brooks, J. O., Switzer, F. S., & Gugerty, L. (2003). Effects of
situation awareness training on novice process control plant
operators. Proceedings of the Human Factors and Ergonomics
Society Annual Meeting, 47, 606–609.
Burns, C. M., Skraaning, G., Jr., Jamieson, G. A., Lau, N., Kwok,
J., Welch, R., & Andresen, G. (2008). Evaluation of ecological
interface design for nuclear process control: Situation aware-
ness effects. Human Factors, 50, 663–679.
Bushman, B. J. (1994). Vote-counting procedures in meta-analysis.
In H. Cooper & L. V. Hedges (Eds.), The handbook of research
synthesis (pp. 192–213). New York, NY: Russell Sage.
Cak, S., Say, B., & Misirlisoy, M. (2019). Effects of working mem-
ory, attention, and expertise on pilots’ situation awareness.
Cognition, Technology & Work. Advance online publication.
doi:10.1007/s10111-019-00551-w
Chiappe, D. L., Rorie, R. C., Moran, C. A., & Vu, K. P. L. (2012).
A situated approach to the acquisition of shared SA in team
contexts. Theoretical Issues in Ergonomics Science, 15, 69–87.
Chiappe, D. L., Strybel, T. Z., & Vu, K. P. L. (2015). A situated
approach to the understanding of dynamic situations. Journal
of Cognitive Engineering and Decision Making, 9, 33–43.
Cohen, J. (1995). Statistical power analysis for the behavioral sci-
ences. New York, NY: Routledge Academic.
Cooke, N. J., Kiekel, P. A., & Helm, E. E. (2001). Measuring team
knowledge during skill acquisition of a complex task. Interna-
tional Journal of Cognitive Ergonomics, 5, 297–315.
Cooke, N. J., Stout, R., Rivera, K., & Salas, E. (1998). Explor-
ing measures of team knowledge. Proceedings of the Human
Factors and Ergonomics Society Annual Meeting, 1, 215–219.
Coolen, E., Draaisma, J., & Loeffen, J. (2019). Measuring situa-
tion awareness and team effectiveness in pediatric acute care
by using the situation global assessment technique. European
Journal of Pediatrics, 178, 837–850.
Cowan, N. (1988). Evolving conceptions of memory storage,
selective attention, and their mutual constraints within the
human information processing system. Psychological Bulletin,
104, 163–191.
Cramer, H. (1946). Mathematical methods of statistics. Princeton,
NJ: Princeton University Press.
Crozier, M. S., Ting, H. Y., Boone, D. C., O’Regan, N. B., Ban-
drauk, N., Furey, A., . . . Hogan, M. P. (2015). Use of human
patient simulation and validation of the Team Situation Aware-
ness Global Assessment Technique (TSAGAT): A multidisci-
plinary team assessment tool in trauma education. Journal of
Surgical Education, 72, 156–163.
Cuevas, H. M., & Bolstad, C. A. (2010). Influence of team leaders’
situation awareness on their team’s situation awareness and
performance. Proceedings of the Human Factors and Ergo-
nomics Society Annual Meeting, 54, 309–313.
Cumming, G. (2014). The new statistics: Why and how. Psycho-
logical Science, 25, 7–29.
Cummings, M. L., & Guerlain, S. (2007). Developing operator
capacity estimates for supervisory control of autonomous vehi-
cles. Human Factors, 49, 1–15.
Cunningham, J. C., Battiste, H., Curtis, S., Hallett, E. C., Koltz, M.,
Brandt, S. L., & Johnson, W. W. (2015). Measuring situation
awareness with probe questions: Reasons for not answering the
probes. Procedia Manufacturing, 3, 2982–2989.
de Winter, J., Eisma, Y., Cabrall, C., Hancock, P., & Stanton, N. A.
(2019). Situation awareness based on eye movements in rela-
tion to the task environment. Cognition, Technology & Work,
21, 99–111.
Durso, F. T., Bleckley, M. K., & Dattel, A. R. (2006). Does situa-
tion awareness add to the validity of cognitive tests? Human
Factors, 48, 721–733.
24 Month XXXX - Human Factors
Durso, F. T., & Gronlund, S. D. (1999). Situation awareness.
In F. T. Durso, R. Nickerson, R. Schvaneveldt, S. Dumais,
S. Lindsay, & M. Chi (Eds.), Handbook of applied cognition
(pp. 284–314). New York, NY: John Wiley.
Durso, F. T., Hackworth, C. A., Truitt, T. R., Crutchfield, J.,
Nikolic, D., & Manning, C. A. (1998). Situation awareness as a
predictor of performance for en route air traffic controllers. Air
Traffic Control Quarterly, 6(1), 1–20.
Endsley, M. R. (1988). Design and evaluation for situation aware-
ness enhancement. Proceedings of the Human Factors Society
Annual Meeting, 32, 97–101.
Endsley, M. R. (1990a). A methodology for the objective mea-
surement of situation awareness. In Situational awareness in
aerospace operations (pp. 1/1–1/9). Neuilly Sur Seine, France:
NATO-AGARD.
Endsley, M. R. (1990b). Situation awareness in dynamic human
decision making: Theory and measurement. Los Angeles: Uni-
versity of Southern California.
Endsley, M. R. (1993a). Situation awareness and workload: Flip
sides of the same coin. In R. S. Jensen & D. Neumeister (Eds.),
Proceedings of the Seventh International Symposium on Avia-
tion Psychology (pp. 906–911). Columbus, OH: Department of
Aviation, The Ohio State University.
Endsley, M. R. (1993b). A survey of situation awareness require-
ments in air-to-air combat fighters. International Journal of
Aviation Psychology, 3, 157–168.
Endsley, M. R. (1995a). Direct measurement of situation aware-
ness in simulations of dynamic systems: Validity and use of
SAGAT. In D. J. Garland & M. R. Endsley (Eds.), Experi-
mental analysis and measurement of situation awareness
(pp. 107–113). Daytona Beach, FL: Embry-Riddle University.
Endsley, M. R. (1995b). Measurement of situation awareness in
dynamic systems. Human Factors, 37, 65–84.
Endsley, M. R. (1995c). Toward a theory of situation awareness in
dynamic systems. Human Factors, 37, 32–64.
Endsley, M. R. (2000). Direct measurement of situation aware-
ness: Validity and use of SAGAT. In M. R. Endsley & D. J.
Garland (Eds.), Situation awareness analysis and measurement
(pp. 147–174). Mahwah, NJ: Lawrence Erlbaum.
Endsley, M. R. (2015a). Final reflections: Situation awareness
models and measures. Journal of Cognitive Engineering and
Decision Making, 9, 101–111.
Endsley, M. R. (2015b). Situation awareness misconceptions and
misunderstandings. Journal of Cognitive Engineering and
Decision Making, 9, 4–32.
Endsley, M. R. (in press). The divergence of objective and subjec-
tive situation awareness: A meta-analysis. Journal of Cognitive
Engineering and Decision Making.
Endsley, M. R., & Bolstad, C. A. (1994). Individual differences
in pilot situation awareness. International Journal of Aviation
Psychology, 4, 241–264.
Endsley, M. R., & Garland, D. J. (Eds.). (2000). Situation awareness
analysis and measurement. Mahwah, NJ: Lawrence Erlbaum.
Endsley, M. R., & Jones, D. G. (1995). Situation awareness
requirements analysis for TRACON air traffic control (TTU-
IE-95-01). Lubbock: Texas Tech University.
Endsley, M. R., & Jones, D. G. (2012). Designing for situation
awareness: An approach to human-centered design (2nd ed.).
London, England: Taylor & Francis.
Endsley, M. R., Mogford, R. H., & Stein, E. S. (1997). Controller
situation awareness in free flight. Proceedings of the Human
Factors and Ergonomics Society Annual Meeting, 41, 4–8.
Endsley, M. R., & Rodgers, M. D. (1994a). Situation awareness
information requirements for en route air traffic control (DOT/
FAA/AM-94/27). Washington, DC: Federal Aviation Adminis-
tration Office of Aviation Medicine.
Endsley, M. R., & Rodgers, M. D. (1994b). Situation aware-
ness information requirements for en route air traffic control.
Proceedings of the Human Factors and Ergonomics Society
Annual Meeting, 38, 71–75.
Endsley, M. R., & Rodgers, M. D. (1998). Distribution of attention,
situation awareness, and workload in a passive air traffic con-
trol task: Implications for operational errors and automation.
Air Traffic Control Quarterly, 6, 21–44.
Endsley, M. R., Selcon, S. J., Hardiman, T. D., & Croft, D. G.
(1998). A comparative evaluation of SAGAT and SART for
evaluations of situation awareness. Proceedings of the Human
Factors and Ergonomics Society Annual Meeting, 42, 82–86.
Endsley, M. R., Sollenberger, R., & Stein, E. (1999). The use of
predictive displays for aiding controller situation awareness.
Proceedings of the Human Factors and Ergonomics Society
Annual Meeting, 43, 51–55.
Endsley, M. R., Sollenberger, R., & Stein, E. (2000). Situation
awareness: A comparison of measures. In Proceedings of the
Human Performance, Situation Awareness and Automation:
User-Centered Design for the New Millennium (pp. 15–19).
Savannah, GA: SA Technologies, Inc.
Farley, T. C., Hansman, R. J., Amonlirdviman, K., & Endsley, M.
R. (2000). Shared information between pilots and controllers
in tactical air traffic control. Journal of Guidance, Control and
Dynamics, 23, 826–836.
Gardner, A. K., Kosemund, M., & Martinez, J. (2017). Examin-
ing the feasibility and predictive validity of the SAGAT tool to
assess situation awareness among medical trainees. Simulation
in Healthcare, 12, 17–21.
Gartenberg, D., Breslow, L., McCurry, J. M., & Trafton, J. G.
(2014). Situation awareness recovery. Human Factors, 56,
710–727.
Golightly, D., Wilson, J. R., Lowe, E., & Sharples, S. (2010). The
role of situation awareness for understanding signalling and
control in rail operations. Theoretical Issues in Ergonomics
Science, 11, 84–98.
Gonzalez, C., & Wimisberg, J. (2007). Situation awareness in
dynamic decision making: Effects of practice and working
memory. Journal of Cognitive Engineering and Decision Mak-
ing, 1, 56–74.
Gorman, J. C., Cooke, N. J., Pederson, H. K., Connor, O. O., &
DeJoode, J. A. (2005). Coordinated awareness of situation by
teams (CAST): Measuring team situation awareness of a com-
munications glitch. Proceedings of the Human Factors and
Ergonomics Society Annual Meeting, 49, 274–277.
Gugerty, L. (1998). Evidence from a partial report task for forget-
ting in dynamic spatial memory. Human Factors, 40, 498–508.
Gugerty, L. J. (1997). Situation awareness during driving: Explicit
and implicit knowledge in dynamic spatial memory. Journal of
Experimental Psychology: Applied, 3, 42–66.
Gutzwiller, R. S., & Clegg, B. A. (2012). The role of working
memory in levels of situation awareness. Journal of Cognitive
Engineering and Decision Making, 7, 141–154.
Hall, J., & Phelps, R. H. (1983). Nonlaboratory techniques for the
study of cognitive and decision-making processes: A descrip-
tion and selected bibliography (No. 83-45). Washington, DC:
U.S. Army Research Institute for the Behavioral and Social
Sciences.
Evaluation of Objective Measures of SA 25
Hallbert, B. P. (1997). Situation awareness and operator perfor-
mance: Results from simulator-based studies. In Proceedings
of the IEEE Sixth Conference on Human Factors and Power
Plants (pp. 18/11–18/16). New York, NY: IEEE.
Hamilton, K., Mancuso, V., Mohammed, S., Tesler, R., & McNeese,
M. (2017). Skilled and unaware: The interactive effects of team
cognition, team metacognition, and task confidence on team
performance. Journal of Cognitive Engineering and Decision
Making, 11, 382–395.
Hauss, Y., & Eyferth, K. (2003). Securing future ATM-concepts’
safety by measuring situation awareness in ATC. Aerospace
Science and Technology, 7, 417–427.
Hogan, M. P., Pace, D. E., Hapgood, J., & Boone, D. C. (2006). Use
of human patient simulation and the situation awareness global
assessment technique in practical trauma skills assessment.
Journal of Trauma and Acute Care Surgery, 61, 1047–1052.
Hogg, D. N., Follesø, K., Strand-Volden, F., & Torralba, B. (1995).
Development of a situation awareness measure to evaluate
advanced alarm systems in nuclear power plant control rooms.
Ergonomics, 38, 2394–2413.
Ikuma, L. H., Harvey, C., Taylor, C. F., & Handal, C. (2014). A
guide for assessing control room operator performance using
speed and accuracy, perceived workload, situation awareness,
and eye tracking. Journal of Loss Prevention in the Process
Industries, 32, 454–465.
Jackson, L., Chapman, P., & Crundall, D. (2009). What happens
next? Predicting other road users’ behaviour as a function of driv-
ing experience and processing time. Ergonomics, 52, 154–164.
Javed, Y., Norris, T., & Johnston, D. (2012, April). Evaluating
SAVER: Measuring shared and team situation awareness of
emergency decision makers. Paper presented at the 9th Interna-
tional ISCRAM Conference, Vancouver, BC, Canada.
Jeannot, E., Kelly, C., & Thompson, D. (2003). The development of
situation awareness measures in ATM systems. Brussels, Bel-
gium: Eurocontrol.
Jipp, M., & Ackerman, P. L. (2016). The impact of higher levels of
automation on performance and situation awareness: A func-
tion of information-processing ability and working-memory
capacity. Journal of Cognitive Engineering and Decision Mak-
ing, 10, 138–166.
Jones, D. G., & Endsley, M. R. (2000a). Can real-time probes pro-
vide a valid measure of situation awareness? In Proceedings
of the Human Performance, Situation Awareness and Automa-
tion: User-Centered Design for the New Millennium Confer-
ence (pp. 245–250). Savannah, GA: SA Technologies.
Jones, D. G., & Endsley, M. R. (2000b). Overcoming represen-
tational errors in complex environments. Human Factors, 42,
367–378.
Jones, D. G., & Endsley, M. R. (2004). Using real time probes for
measuring situation awareness. International Journal of Avia-
tion Psychology, 14, 343–367.
Kane, M. J., Bleckley, M. K., Conway, A. R., & Engle, R. W.
(2001). A controlled-attention view of working-memory
capacity. Journal of Experimental Psychology: General, 130,
169–183.
Kass, S. J., Cole, K. S., & Stanny, C. J. (2007). Effects of distrac-
tion and experience on situation awareness and simulated driv-
ing. Transportation Research, Part F: Traffic Psychology and
Behaviour, 10, 321–329.
Keeler, J., Battiste, H., Hallett, E. C., Roberts, Z., Winter, A., San-
chez, K., . . . Vu, K.-P. L. (2015). May I interrupt? The effect
of SPAM probe questions on air traffic controller performance.
Procedia Manufacturing, 3, 2998–3004.
Kennedy, J. E. (2015). Critique of Cumming’s “new statistics” for
psychological research: A perspective from outside psychol-
ogy. Retrieved from https://jeksite.org/psi/critique_new_stat
.pdf
Kline, R. D. (2004). Beyond significance testing: Reforming data
analysis methods in behavioral research. Washington, DC:
APA Books.
Kraemer, J., & Süß, H. M. (2015). Real time validation of online
situation awareness questionnaires in simulated approach air
traffic control. Procedia Manufacturing, 3, 3152–3159.
Landry, S. J., & Yoo, H.-S. (2012). Sampling error and other statis-
tical problems with query-based situation awareness measures.
Proceedings of the Human Factors and Ergonomics Society
Annual Meeting, 56, 292–296.
Langan-Fox, J., Sankey, M. J., & Canty, J. M. (2009). Human fac-
tors measurement for future air traffic control systems. Human
Factors, 51, 595–637.
Lau, N., Jamieson, G. A., & Skraaning, G. (2014). Inter-rater reli-
ability of query/probe-based techniques for measuring situa-
tion awareness. Ergonomics, 57, 959–972.
Lau, N., Jamieson, G. A., & Skraaning, G., Jr. (2016). Situation
awareness acquired from monitoring process plants—The
Process Overview concept and measure. Ergonomics, 59,
976–988.
Leggatt, A. (2004, September 14–16). Objectively measuring the
promulgation of commander’s intent in a coalition effects
based planning experiment (MNE3). Proceedings of the 9th
International Command and Control Research and Technology
Symposium, Copenhagen, Denmark.
Lo, J. C., Sehic, E., Brookhuis, K. A., & Meijer, S. A. (2016).
Explicit or implicit situation awareness? Measuring the situ-
ation awareness of train traffic controllers. Transportation
Research, Part F: Traffic Psychology and Behaviour, 43,
325–338.
Loft, S., Bowden, V., Braithwaite, J., Morrell, D. B., Huf, S., &
Durso, F. T. (2015). Situation awareness measures for simu-
lated submarine track management. Human Factors, 57, 298–
310.
Loft, S., Jooste, L., Li, Y. R., Ballard, T., Huf, S., Lipp, O. V., &
Visser, T. A. (2018). Using situation awareness and workload
to predict performance in submarine track management: A
multilevel approach. Human Factors, 60, 978–991.
Loft, S., Morrell, D. B., Ponton, K., Braithwaite, J., Bowden, V.,
& Huf, S. (2016). The impact of uncertain contact location on
situation awareness and performance in simulated submarine
track management. Human Factors, 58, 1052–1068.
Loft, S., Sadler, A., Braithwaite, J., & Huf, S. (2015). The chronic
detrimental impact of interruptions in a simulated submarine
track management task. Human Factors, 57, 1417–1426.
McGowan, A., & Banbury, S. (2004). Interruption and reorienta-
tion effects of a situation awareness probe on driving hazard
anticipation. Proceedings of the Human Factors and Ergonom-
ics Society Annual Meeting, 48, 290–294.
McGuinness, B. (2004). Quantitative analysis of situational aware-
ness (QUASA): Applying signal detection theory to true/false
probes and self-ratings. In Proceedings of the Command and
Control Research and Technology Symposium (pp. 1–14). Bris-
tol, UK: BAE Systems Advanced Technology Centre.
Morgan, C. A., Chiappe, D. L., Kraut, J., Strybel, T. Z., & Vu,
K. P. L. (2012). An investigation into the potential sources
of interference in SA probe techniques. In S. J. Landry (Ed.),
Advances in human aspects of aviation (pp. 623–631). Boca
Raton, FL: CRC Press.
26 Month XXXX - Human Factors
Morgan, P., Tregunno, D., Brydges, R., Pittini, R., Tarshis, J.,
Kurrek, M., . . . Ryzynski, A. (2015). Using a situational
awareness global assessment technique for interprofessional
obstetrical team training with high fidelity simulation. Journal
of Interprofessional Care, 29, 13–19.
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can
know: Verbal reports on mental processes. Psychological
Review, 84, 231–259.
Orasanu, J. (2000). Evaluating team situation awareness through
communication. In D. J. Garland & M. R. Endsley (Eds.),
Experimental analysis and measurement of situation aware-
ness (pp. 283–288). Daytona Beach, FL: Embry-Riddle Uni-
versity.
Parush, A. G., Mastoras, A., Bhandari, A., Momtahan, K., Day, K.,
Weitzman, B., . . . Calder, L. (2017). Can teamwork and situ-
ational awareness (SA) in ED resuscitations be improved with
a technological cognitive aid? Design and a pilot study of a
team situation display. Journal of Biomedical Informatics, 76,
154–161.
Pierce, R. S. (2012). The effect of SPAM administration during a
dynamic simulation. Human Factors, 54, 838–848.
Pierce, R. S., Strybel, T. Z., & Vu, K.-P. L. (2008). Comparing situ-
ation awareness measurement techniques in a low fidelity air
traffic control simulation. In Proceedings of the 26th Interna-
tional Congress of the Aeronautical Sciences (ICAS) (pp. 1–8).
Anchorage, AS: ICAS.
Pierce, R. S., Vu, K.-P. L., Nguyen, J., & Strybel, T. Z. (2008).
The relationship between SPAM, workload, and task per-
formance on a simulated ATC task. Proceedings of the
Human Factors and Ergonomics Society Annual Meeting,
52, 34–38.
Poole, B. J., & Kane, M. J. (2009). Working-memory capacity pre-
dicts the executive control of visual search among distractors:
The influences of sustained and selective attention. The Quar-
terly Journal of Experimental Psychology, 62, 1430–1454.
Price, T. F., & LaFiandra, M. (2017). The perception of team
engagement reduces stress induced situation awareness over-
confidence and risk-taking. Cognitive Systems Research, 46,
52–60.
Prince, C., Ellis, E., Brannick, M. T., & Salas, E. (2007). Mea-
surement of team situation awareness in low experience level
aviators. The International Journal of Aviation Psychology, 17,
41–57.
Prince, C., Salas, E., & Stout, R. (1995). Situation awareness: Team
measures, training methods. In D. J. Garland & M. R. Endsley
(Eds.), Experimental analysis and measurement of situation
awareness (pp. 135–140). Daytona Beach, FL: Embry-Riddle
University Press.
Pritchett, A. R., Hansman, R. J., & Johnson, E. N. (1995). Use
of testable responses for performance-based measurement of
situation awareness. In D. J. Garland & M. R. Endsley (Eds.),
Experimental analysis and measurement of situation aware-
ness (pp. 75–81). Daytona Beach, FL: Embry-Riddle Univer-
sity Press.
Randel, J. M., Pugh, H. L., & Reed, S. K. (1996). Differences in
expert and novice situation awareness in naturalistic decision
making. International Journal of Human-computer Studies,
45, 579–597.
Ratwani, R. M., & Trafton, J. G. (2008). Spatial memory guides
task resumption. Visual Cognition, 16, 1001–1010.
Rose, J., Bearman, C., Naweed, A., & Dorrian, J. (2019). Proceed
with caution: Using verbal protocol analysis to measure situa-
tion awareness. Ergonomics, 62, 115–127.
Rosenman, E. D., Dixon, A. J., Webb, J. M., Brolliar, S., Golden,
S. J., Jones, K. A., . . . Chao, G. T. (2018). A simulation-based
approach to measuring team situational awareness in emer-
gency medicine: A multicenter, observational study. Academic
Emergency Medicine, 25, 196–204.
Sætrevik, B. (2012). A controlled field study of situation aware-
ness measures and heart rate variability in emergency handling
teams. Proceedings of the Human Factors and Ergonomics
Society Annual Meeting, 56, 2006–2010.
Salmon, P. M., Stanton, N., Walker, G., Baber, C., Jenkins, D.,
McMaster, R., & Young, M. (2008). What is really going on?
Review of situation awareness models for individuals and
teams. Theoretical Issues in Ergonomics Science, 9, 297–323.
Salmon, P. M., Stanton, N., Walker, G., & Green, D. (2006). Situa-
tion awareness measurement: A review of applicability for C4i
environments. Applied Ergonomics, 37, 225–238.
Salmon, P. M., Stanton, N. A., & Young, K. L. (2011). Situation
awareness on the road: Review, theoretical and methodological
issues and future directions. Theoretical Issues in Ergonomics
Science, 13, 472–492.
Saner, L. D., Bolstad, C. A., Gonzalez, C., & Cuevas, H. M. (2009).
Measuring and predicting shared situation awareness in teams.
Journal of Cognitive Engineering and Decision Making, 3,
280–308.
Sarter, N. B., & Woods, D. D. (1991). Situation awareness: A criti-
cal but ill-defined phenomenon. The International Journal of
Aviation Psychology, 1, 45–57.
Seet, A. W., Teh, C. A., Soo, J. K., & Teo, L. (2004). Constructible
assessment for situation awareness in a distributed C2 envi-
ronment. Singapore: DSO National Laboratories.
Shelton, C. L., Kinston, R., Molyneux, A. J., & Ambrose, L. J. (2013).
Real-time situation awareness assessment in critical illness man-
agement: Adapting the situation present assessment method to
clinical simulation. BMJ Quality & Safety, 22, 163–167.
Shipstead, Z., Lindsey, D. R., Marshall, R. L., & Engle, R. W.
(2014). The mechanisms of working memory capacity: Pri-
mary memory, secondary memory, and attention control. Jour-
nal of Memory and Language, 72, 116–141.
Silva, H. I., Grigoleit, T., Ann Burress, M., & Fitzpatrick, D.
(2017). Measuring the impact of console operator experi-
ence in a simulated petrochemical refining emergency event.
Proceedings of the Human Factors and Ergonomics Society
Annual Meeting, 61, 527–531.
Silva, H. I., Ziccardi, J., Grigoleit, T., Battiste, V., Strybel, T. Z., &
Vu, K.-P. L. (2013). Are the intrusive effects of spam probes
present when operators differ by skill level and training? In
S. Yamamoto (Ed.), Proceedings of the International Confer-
ence on Human Interface and the Management of Information
(pp. 269–275). Berlin: Springer.
Smolensky, M. W. (1993). Toward the physiological measurement
of situation awareness: The case for eye movement measure-
ments. In Proceedings of the Human Factors and Ergonom-
ics Society 37th Annual Meeting (p. 41). Santa Monica, CA:
Human Factors and Ergonomics Society.
Snow, M. P., & Reising, J. M. (1999). Effect of pathway-in-the-
sky and synthetic terrain imagery on situation awareness in a
simulated low-level ingress scenario. Wright-Patterson AFB,
OH: Air Force Research Lab Crew System Interface Division.
Soliman, A. M., & Mathna, E. K. (2009). Metacognitive strategy
training improves driving situation awareness. Social Behavior
and Personality: An International Journal, 37, 1161–1170.
Sorensen, L. J., Stanton, N., & Banks, A. P. (2010). Back to SA
school: Contrasting three approaches to situation awareness
Evaluation of Objective Measures of SA 27
in the cockpit. Theoretical Issues in Ergonomics Science, 12,
451–471.
Stanton, N. A., Salmon, P. M., Rafferty, L. A., Walker, G. H.,
Baber, C., & Jenkins, D. P. (2017). Human factors methods:
A practical guide for engineering and design. Boca Raton, FL:
CRC Press.
Stanton, N. A., Salmon, P. M., Walker, G., & Jenkins, D. (2010).
Is situation awareness all in the mind? Theoretical Issues in
Ergonomics Science, 11, 29–40.
Strater, L. D., Endsley, M. R., Pleban, R. J., & Matthews, M. D.
(2001). Measures of platoon leader situation awareness in vir-
tual decision making exercises (Research Report No. 1770).
Alexandria, VA: Army Research Institute.
Strybel, T. Z., Chiappe, D., Vu, K.-P. L., Miramontes, A., Battiste,
H., & Battiste, V. (2016). A comparison of methods for assess-
ing situation awareness in current day and future air traffic
management operations: Graphics-based vs text-based online
probe systems. IFAC-PapersOnLine, 49(19), 31–35.
Strybel, T. Z., Vu, K.-P. L., Bacon, L. P., Kraut, J., Battiste, V., &
Johnson, W. (2010). Diagnosticity of an online query technique
for measuring pilot situation awareness in NextGen. In Proceed-
ings of the 2010 IEEE/AIAA 29th Digital Avionics Systems Con-
ference (DASC) (pp. 4. B. 1-1–4. B. 1-12). New York, NY: IEEE.
Strybel, T. Z., Vu, K.-P. L., Battiste, V., & Johnson, W. (2013).
Measuring the impact of NextGen operating concepts for sepa-
ration assurance on pilot situation awareness and workload.
The International Journal of Aviation Psychology, 23, 1–26.
Strybel, T. Z., Vu, K.-P. L., Kraft, J., & Minakata, K. (2008).
Assessing the situation awareness of pilots engaged in self
spacing. Proceedings of the Human Factors and Ergonomics
Society Annual Meeting, 52, 11–15.
Sulistyawati, K., Chui, Y. P., & Wickens, C. D. (2008). Multi-method
approach to team situation awareness. Proceedings of the Human
Factors and Ergonomics Society Annual Meeting, 52, 463–467.
Sulistyawati, K., Wickens, C. D., & Chui, Y. P. (2009). Explor-
ing the concept of team situation awareness in a simulated air
combat environment. Journal of Cognitive Engineering and
Decision Making, 3, 309–330.
Sulistyawati, K., Wickens, C. D., & Chui, Y. P. (2011). Prediction
in situation awareness: Confidence bias and underlying cogni-
tive abilities. International Journal of Aviation Psychology, 21,
153–174.
Sullivan, C., & Blackman, H. S. (1991). Insights into pilot situation
awareness using verbal protocol analysis. Proceedings of the
Human Factors Society Annual Meeting, 35, 57–61.
Taber, M. J., McCabe, J., Klein, R. M., & Pelot, R. P. (2013).
Development and evaluation of an offshore oil and gas Emer-
gency Response Focus Board. International Journal of Indus-
trial Ergonomics, 43, 40–51.
Trapsilawati, F., Wickens, C. D., Qu, X., & Chen, C.-H. (2016).
Benefits of imperfect conflict resolution advisory aids for
future air traffic control. Human Factors, 58, 1007–1019.
Tremblay, S., Vachon, F., Lafond, D., & Kramer, C. (2012). Deal-
ing with task interruptions in complex dynamic environments:
Are two heads better than one? Human Factors, 54, 70–83.
Unsworth, N., & Engle, R. W. (2005). Working memory capacity
and fluid abilities: Examining the correlation between opera-
tion span and Raven. Intelligence, 33, 67–81.
Unsworth, N., & Spillers, G. J. (2010). Working memory capacity:
Attention control, secondary memory, or both? A direct test of
the dual-component model. Journal of Memory and Language,
62, 392–406.
van der Linden, S., & Chryst, B. (2015). Why the “new statistics”
isn’t new. Psychologist, 28, 610.
Vidulich, M. (2000). Testing the sensitivity of situation awareness
metrics in interface evaluations. In M. R. Endsley & D. J. Gar-
land (Eds.), Situation awareness analysis and measurement
(pp. 227–248). Mahwah, NJ: Lawrence Erlbaum.
Vidulich, M. A., & Tsang, P. S. (2015). The confluence of situation
awareness and mental workload for adaptable human-machine
systems. Journal of Cognitive Engineering and Decision Mak-
ing, 9, 95–97.
Walker, G. H., Stanton, N. A., & Young, M. S. (2008). Feedback
and driver situation awareness (SA): A comparison of SA mea-
sures and contexts. Transportation Research Part F: Traffic
Psychology and Behaviour, 11, 282–299.
Wickens, C. D. (1995). The tradeoff of design for routine and unex-
pected performance: Implications of situation awareness. In
D. J. Garland & M. R. Endsley (Eds.), Experimental analysis
and measurement of situation awareness (pp. 57–64). Daytona
Beach, FL: Embry-Riddle Aeronautical University Press.
Wickens, C. D. (1998). Statistics. Ergonomics in Design, 6(4),
18–22.
Wickens, C. D. (2000). The tradeoff of design for routine and unex-
pected performance: Implications of situation awareness. In M.
R. Endsley & D. J. Garland (Eds.), Situation awareness analy-
sis and measurement (pp. 211–226). Mahwah, NJ: Lawrence
Erlbaum.
Willems, B. F., & Heiney, M. (2001, December 3–7). Real-time
assessment of situation awareness of air traffic control special-
ists on operational host computer system and display system
replacement hardware. Proceedings of the USA/Europe Air
Traffic Management R&D Seminar, Santa Fe, NM.
Wilson, G. F. (2000). Strategies for psychophysiological assess-
ment of situation awareness. In M. R. Endsley & D. J. Gar-
land (Eds.), Situation awareness analysis and measurement
(pp. 175–188). Mahwah, NJ: Lawrence Erlbaum.
Yuan, X., She, M., Li, Z., Zhang, Y., & Wu, X. (2016). Mutual
awareness: Enhanced by interface design and improving team
performance in incident diagnosis under computerized work-
ing environment. International Journal of Industrial Ergonom-
ics, 54, 65–72.
Mica Endsley is President of SA Technologies and is
the former Chief Scientist of the U.S. Air Force. She
received a PhD in Industrial and Systems Engineer-
ing from the University of Southern California. She
has published extensively on situation awareness,
automation, and system design.
Date received: February 12, 2019
Date accepted: August 15, 2019