See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/259910733

Measurement of situation awareness for automobile technologies of the future

Chapter · January 2010

CITATIONS READS
14 3,024

5 authors, including:

Cheryl Bolstad Mica R. Endsley

Sandia National Laboratories SA Technologies
66 PUBLICATIONS   1,173 CITATIONS    216 PUBLICATIONS   28,317 CITATIONS   

SEE PROFILE SEE PROFILE

Linda Sala Angell

Touchstone Evaluations, Inc.
76 PUBLICATIONS   770 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Collaboration Tools for the Advanced Decision Architecture View project

Careers View project

All content following this page was uploaded by Cheryl Bolstad on 01 April 2015.

The user has requested enhancement of the downloaded file.

 DVM- 11

Measurement of Situation Awareness for Automobile Technologies of the Future

Cheryl A. Bolstad, Haydee M. Cuevas, Jingjing Wang-Costello, and Mica R. Endsley

SA Technologies, Inc.

Linda S. Angell
Touchstone Evaluations, Inc.

Copyright © 2009 SAE International

ABSTRACT
We present a theoretical model of situation awareness
(SA) to highlight how driver SA may be influenced by
characteristics and limitations of both the human driver
(e.g., attention, memory) as well as the technologies that
have been increasingly introduced into private and
commercial vehicles. Such technologies include devices
that are integrated with the vehicle, such as navigation
and telematics systems, as well as those that are carried
in by drivers, such as cell phones, PDAs, etc., and also
include more advanced automation technologies, such
as adaptive cruise control, lane centering systems,
autonomous braking, and other systems. We then
provide a brief overview of the different approaches for
measuring SA, citing the advantages and disadvantages
of each approach. These measurement techniques are
presented as guidelines for automotive researchers and
others interested in assessing SA.
INTRODUCTION
Driving a car is a common activity that many of us take
for granted. However, safe and effective driving
performance requires the ability to attend to important
information, process this information, and then react
accordingly. This complex and cognitively challenging
ability is encompassed within the construct of situation
awareness (SA). SA can be broadly defined as the
internal conceptualization of the current situation [1].
More formally, SA involves “…the perception of the
elements in the environment within a volume of time and
space, the comprehension of their meaning and the
projection of their status in the near future” [2, p.36]. For
many years, SA has been primarily associated with
aviation and military performance. Yet, it is now
considered to be a central organizing construct that
forms a key foundation for effective decision making and
human performance in a wide variety of dynamic
systems and challenging environments [3].
One could argue that the driving domain is such a
dynamic system consisting of human, technological, and
environmental elements [4]. A driver must be aware of
road conditions, the presence of other cars or obstacles,
and the rate of change of traffic in the area in order to
predict whether it is safe to change lanes or to pass
another car. Thus, important research questions that
must be addressed by safety researchers and
automobile developers include, for example, determining
if drivers are capable of perceiving and comprehending
all that is occurring around them, and if they are able to
guide their future actions based on their understanding
of the current situation. In addition to these primary tasks
of driving, secondary devices used in the vehicle can
further complicate the driving task and have the potential
to impact SA. These devices are of two broad types:
those that are integrated with the vehicle (e.g., built-in
navigation or telematics systems) and those that are
carried into the vehicle by drivers (e.g., cell phones,
PDAs). The introduction of advanced automation
technologies (e.g., adaptive cruise control, lane
centering systems, autonomous braking systems) may
hold still further implications for SA. Accordingly, a better
understanding of how SA is influenced by both human
and technological characteristics and limitations can

The Engineering Meetings Board has approved this paper for publication. It has successfully completed SAE’s peer review process under the supervision of the
session organizer. This process requires a minimum of three (3) reviews by industry experts.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic,
mechanical, photocopying, recording, or otherwise, without the prior written permission of SAE.
ISSN 0148-7191
Positions and opinions advanced in this paper are those of the author(s) and not necessarily those of SAE. The author is solely responsible for the content of
the paper.
SAE Customer Service: Tel: 877-606-7323 (inside USA and Canada)
Tel: 724-776-4970 (outside USA)
Fax: 724-776-0790
Email: CustomerService@sae.org
SAE Web Address: http://www.sae.org
Printed in USA
guide the design of systems (e.g., cars, equipment, etc.)
and intervention techniques (e.g., driver safety
programs, infrastructure changes) to improve driving
performance.
Next, we present a theoretical model of SA to explain
how SA is formed and also discuss how technology may
impact driver SA. We then provide a brief overview of the
different approaches for measuring SA, citing the
advantages and disadvantages of each approach. These
measurement techniques are presented as guidelines for
automotive researchers and others interested in
assessing SA. We conclude with directions for future
research.
MODEL OF SITUATION AWARENESS
A considerable body of research supports the notion that
SA is formed through several steps or stages [2, 5, 6, 7].
As noted earlier, Endsley’s [2] definition of SA identifies
three distinct levels: perception, comprehension and
projection. Perception (Level 1 SA) involves the sensory
detection of significant environmental cues. Perception is
an active process whereby individuals extract salient
cues from their environment. By selectively directing
attention to the incoming stimuli, the individual attends to
important information, such as speed and location, while
disregarding nonessential items. Comprehension (Level
2 SA) involves integrating this information in working
memory to understand how the information will impact
the individual’s goals and objectives [7]. This involves
combining separate pieces of information together to
form a comprehensive picture of the world, or of that
portion of the world of concern to the individual, such as
a change in car direction or sudden loss of speed.
Projection (Level 3 SA) involves extrapolating this
information forward in time to determine how it will affect
future states of the operating environment [8, 9]. Level 3
SA combines what the individual knows about the
current situation with his/her mental model or schemata
of similar events to predict what might happen next. For
instance, a driver approaching a red light might predict
that ignoring the light and proceeding into the
intersection would result in a crash based on his/her
prior driving experience. Thus, mental models and
schemata help ease the cognitive workload required to
develop Levels 2 and 3 SA. These cognitive processes
all form part of the larger human information-processing
model in which SA is used to make decisions and is, in
turn, reflected in subsequent actions (see Figure 1). As
will be discussed next, applying this model to driving can
highlight the criticality of SA to this common albeit
inherently complex task.
Figure 1: Theoretical model of situation awareness (adapted from [2]).
SITUATION AWARENESS IN DRIVING
The construct of SA in dynamic systems applies readily
to the complex task of driving. For example, Ma and
Kaber [4] argued that driving tasks generally involve five
time-phased information processing functions, including
perception, comprehension, and projection, as well as a
decision on a course of action and implementing the
action. In their studies, they observed that all levels of
SA (perception, comprehension, and projection) appear
to have an impact on operational driving behaviors in
following tasks, and consequently, performance. Further,
both Ward [10] and Matthews, Bryant, Webb, and
Harbluk [11] related three general types of driving tasks,
including operational, tactical, and strategic tasks, to the
three levels of SA. These research findings fit well with
the model of SA illustrated in Figure 1. Specific
examples of how the three levels of SA can be applied to
the driving task are shown in Figure 2.
Offering a slightly different perspective, Matthews et al.
[11] outlined multiple elements of awareness defining SA
in driving, including spatial awareness (appreciation of
the location of all relevant features of the environment),
identity awareness (knowledge of salient items in the
driving environment), temporal awareness (knowledge of
Figure 2: Illustration of three levels of situation awareness in driving.
the changing spatial ‘picture’ over time), goal awareness
(driver’s intention of navigating to the destination, and
the maintenance of speed and direction), and system
awareness (relevant information on the vehicle within the
driving environment, which may also be viewed as a
system). In-vehicle system interaction knowledge has
also been shown to be important in the driving
environment. For example, when a car traveling at a
constant speed under cruise control enters a higher
speed limit area, driver awareness of their vehicle
speed, the speed limit, and knowledge of how to set a
higher speed all represent good SA [11, 12].
Other studies have also shown the criticality of SA in
driver performance. Indeed, poor SA has often been
implicated in vehicle crashes [13]. For example, two of
the leading causes of driving accidents, ‘improper
lookout’ and ‘inattention,’ might be thought of as failures
to maintain SA [14]. Often, drivers fail to ‘look ahead’
(and behind) and to keep track of traffic and
environmental changes. Moreover, in-depth studies of
crash causation have highlighted attentional and
perceptual errors (two processes underlying Level 1 SA)
as important direct causes of road accidents [15]. Next,
we discuss several examples of how technology can
potentially impact driver SA.
SITUATION AWARENESS AND AUTOMATION
TECHNOLOGY
Although the increasing use of automation in
automobiles, such as adaptive cruise control and
autonomous steering and braking systems, may offer
tremendous promise to improve driver SA and
performance, such technological aids are not without
limitations and may even present potential unintended
risks to safety. These technologies are designed for
different purposes, depending on the specific system. In
some cases, automation and advanced technology is
applied to enhance driver-vehicle safety; in other
instances, systems are intended to improve convenience
and comfort for drivers. However, whatever their
purpose, such technologies may also generate
unintended negative effects on driver behavior, including
increasing the monitoring workload of the driver and
distracting attention away from driving task performance.
As evident in other complex domains, automation can
change the nature of the demands and responsibilities
on the operator, often in ways that were unintended or
unanticipated by the designer [16]. For example,
automation can hinder SA by creating an out-of-the-loop
experience for operators [3]. In such situations,
operators will have difficulty intervening if they need to
take over or override the automation [17]. However, not
all automation is bad; in fact, automation, when well
designed, can help reduce the complexities of the driving
task. It is in recognition of these issues that so much
research has been initiated and is underway on these
automotive technologies – namely, to identify and
accurately quantify the effects of these systems.
One notable example of automation technology
integrated into vehicles is adaptive cruise control (ACC).
ACC automates one part of the driving task (namely,
controlling headway to a lead vehicle) by adjusting
vehicle speed based upon algorithms that typically use a
defined minimum following distance and information
obtained from monitoring traffic – and which may lead to
the instigation of braking or acceleration when the
detected headway is smaller or larger than the set
criteria [4]. ACC has been shown to improve driving
safety by reducing instances of unsafe headway
distance in following tasks, thereby reducing the
frequency of ‘tailgating’ and the severity of rear-end
collisions [10]. Setting ACC to a long headway distance
(e.g., 2.4s) has also been shown to significantly lower
driving workload, as compared to driving without ACC.
However, the use of ACC in one study was shown to
present potential disadvantages, notably in terms of the
secondary effects of reduced attention on driver SA.
Ward [10] observed that the use of ACC led to poorer
attention to lane positioning and failure to yield to traffic
as well as slower response times to unexpected events.
Another study also showed that drivers may direct their
attention away from driving while using ACC and that the
use of ACC may lead to unexpected increases in
crashes from driver distraction when performing more in-
vehicle secondary tasks [18]. As such, one might
hypothesize that the use of other technologies, such as
cell phones, while driving may diminish the benefits
offered by ACC [4].
However, the results of another study on ACC warrant
consideration in this context. The Automotive Collision
Avoidance System (ACAS) Field Operational Trial (FOT)
was the largest operational test within the USDOT
Intelligent Vehicle Initiative at the time it was conducted.
It was carried out through a partnership between
USDOT, General Motors, and others over a 5-year
period. ACC was examined in the context of a Forward
Collision Warning (FCW) technology in a small fleet of
instrumented vehicles equipped with these advanced
systems. Participants in the study were given a vehicle
from the fleet to drive as their own for a period of 3 to 4
weeks. Several key findings were noted by Kiefer and
Salinger [19] in 2006, at the conclusion of the ACAS
FOT. ACC reduced tailgating behavior, increased lane
dwelling, and led to no unintended safety consequences.
Kiefer and Salinger [19] reported that the ACC system
was found to be broadly used, readily learned, and
attentively supervised by its drivers (including over
relatively lengthy periods of sustained engagement and
vehicle-following). Responses to events (e.g., cut-ins)
were characterized by short reaction times and modest
manual decelerations). No pattern of dependence on the
braking portion of the system could be discerned. FCW
also reduced tailgating behavior, produced valuable
alerts, led to no unintended safety consequences, but
also produced no broad effect on “closing conflicts.” With
respect to engagement in secondary tasks, there was no
statistically significant difference in behaviors between
manual and ACC-assisted driving; secondary tasks were
engaged in about 20% of driving time for both manual
and ACC-assisted conditions. Data for comparison with
use of conventional cruise control were sparse. Similarly,
under conditions of FCW only, frequency and pattern of
secondary task engagement was largely the same, even
across weeks of use (with a slight increase in
conversation with passengers in week 2 only).
This set of studies demonstrates both the importance
and the complexity of examining SA effects that may
arise from the introduction of advanced technologies and
automation. It illustrates the potential value of having
metrics for each of the levels of SA, in building an
understanding of how a system affects the driver-vehicle
system. And these studies also illustrate that knowledge
builds across studies, and requires many, many studies
– at least some of them involving naturalistic driving over
an extended period –before a full picture begins to
emerge, since there is variability between drivers,
between system implementations, and between driving
conditions and maneuvers.
SITUATION AWARENESS AND IN-VEHICLE
SECONDARY SYSTEMS
In-vehicle secondary systems such as navigation aids
based on Global Positioning Systems (GPS) potentially
improve driver performance by supporting navigation
tasks during driving, particularly in unfamiliar
environments. In models of driving, navigation and route-
following are often regarded as part-and-parcel of the
driving task – at the strategic and tactical levels (e.g.,
[20]). In such models, the strategic level would include
activities related to determining trip goals and planning a
route to a destination, whereas the tactical level of
driving allows for drivers to maneuver through existing
circumstances on the road, and the operational level
provides for moment-to-moment operation of the vehicle.
However, when a navigation aid is available to support
the driver in route selection and route following (whether
it is a paper map, or an electronic display embedded in
the vehicle, or a hand-held unit carried into the vehicle),
it can represent a secondary task (depending on its
implementation) that must be time-shared with the
primary task of controlling the vehicle. As such, some
implementations of navigation aids have the potential to
decrease driver attention to the roadway and SA,
particularly Level 1 SA, which, in turn, may lead to
performance problems [21]. For example, the use of
paper maps by drivers has been studied, and difficulties
of doing so documented in comparison to alternatives
(e.g., [22]). Navigation aids may facilitate achieving a
high level of SA for navigation tasks but may
inadvertently undermine attention to operational or
tactical driving behaviors (e.g., braking, passing, etc.) at
certain points in the driving task [21]. Increased
demands on visual attention allocation to a navigation
aid that has chosen to display a visually-intensive
moving map screen may negatively impact drivers’
ability to achieve SA of the external driving environment
– yet a different implementation (such as voiced route
guidance instructions) may not. In addition, the reliability
and accuracy of navigation systems in providing
information support to the driver may differentially
influence driver SA and possibly driver performance in
terms of strategic behaviors. More specifically, whereas
higher reliability leads to higher driver SA, systematic
decreases in navigation aid reliability has been shown to
decrease SA by diverting driver attention from the
roadway [21].
One of the most notable programs of research on
navigation aids was the Travtek program [22]. In one of
its studies, an instrumented car was used to obtain a
comprehensive set of measures about driving
performance and behavior. Six types of navigation/route-
guidance implementations were examined: (visual) map
display alone, (visual) map display with voiced guidance
instructions, (visual) symbolic guidance display alone,
(visual) symbolic guidance display with voiced guidance
instructions, paper map, and paper text instructions. One
of the most important findings to have emerged was that
turn-by-turn guidance information enhanced
performance, usability, and safety. Among the electronic
systems tested, the turn-by-turn display with voiced
guidance instructions led to the best overall
performance. The paper map proved least usable.
The performance measures taken were extensive,
setting a high standard of excellence for on-road studies.
They included measures of lateral control, longitudinal
control, visual scanning, and navigation performance
(from time to plan a trip, drive a trip, total trip time,
number of stops, time stopped, and number of errors).
With the current availability of SA measures, it would be
interesting in future studies of navigation and route
guidance to also gather data on Level 2 and 3 SA, where
certain system features may manifest themselves as
especially beneficial – and others less so.
SITUATION AWARENESS AND CARRIED-IN
PORTABLE DEVICES
One portable device that is commonly carried into the
vehicle by drivers and used while on the road is the cell
phone. It comes in many types – from the simple
handheld, to handheld with hands-free features,
including headsets, to completely hands-free (fully
embedded hands-free is not typically portable or carried-
in, but installed in-vehicle).
The controversy regarding the effects of cell phone tasks
on driving – both in the scientific literature and in the
public policy arena – largely focuses on issues centering
on the extent to which phoning tasks affect SA while
driving. The evidence in the literature contains conflicting
evidence on this point – which is not surprising in light of
the variation in devices and the variation in driving
conditions that have been examined. Further, while a
variety of methods have been employed to examine this
question, it does not appear that formal SA
measurement techniques (of the type described later)
have yet been applied.
A brief review of the literature reveals that an early
epidemiological and statistical analyses of crash data
estimated that drivers with cell phones were four times
as likely to be involved in a crash -- and made the claim
that hands-free telephones provided no safety benefit
(though the variety of hands-free phones studied did not
include any fully-embedded systems) [23, 24].
Subsequent studies [25, 26, 27] addressed some of the
methodological issues of that first early epidemiological
study, and have produced findings that contrast with the
early application of the case-crossover method. For
example, Young and Schreiner [26] examined personal
conversations which were carried out using a hands-free
embedded device (which also allowed precise time
relationships to be established) and found the risk of an
airbag crash was somewhere in a range from a
moderately lower risk to a risk near that of driving conversation tasks had in studies done on the road –
without a recent personal conversation. These results they correspond well – and it furthermore becomes
are not consistent with the large increase in crash risk apparent that miss rates and reaction times for
reported in using the case-crossover method. Hahn and conversation of the type studied in these experiments
Prieger [27] used data on number of crashes together leads to much smaller effects on event detection miss
with data from a national Internet survey on cell phone rates and reaction times than do visual-manual tasks
use to develop a regression model estimating the effects (which produced miss rates in the range of 24-40% and
of cell phone use on crash risk. Their analysis indicated slowed reaction times by ~200 ms or more relative to
that the effect of cell phone use on crashes varied “just driving” for events occurring in the driver’s forward
across segments of the population. When corrections for view) [33].
self-selection bias and endogeneity were made (notions
related to the idea that drivers who use the cell phone Thus, given the landscape of findings on phoning and
while driving may also be inherently less safe drivers conversation tasks, it is interesting to consider whether
even when not on the phone), there was no significant research which would apply measures developed
effect on risk for hand-held or hands-free phone use. specifically to examine SA would bring new insights to
this body of work (and to the larger body of work on
This issue has also been studied from the perspective of
device use beyond cell phones). Such a research
driving performance (e.g., in simulators, on test tracks,
enterprise may well illuminate new dimensions of these
and on-the-road). Several studies [28, 29] have indicated
issues, and extend the state of knowledge.
that drivers exhibit slower responses to traffic light
changes in the presence of an experimentally-produced
SUMMARY - The preceding discussion, although brief,
in-vehicle distracter (artificial memory tasks performed
highlights how automation technologies and other
using a touchpad that looked like a telephone keypad),
devices used in the vehicle (whether integrated with the
and have shown that tasks with a visual-manual
vehicle or carried into the vehicle) may sometimes
component can have effects at crucial points in a driving
negatively impact driver SA through the introduction of
maneuver using basic automatic transmission cars. In
‘SA Demons’ (i.e., factors that work to undermine SA).
another study, it was shown that in comparison to
These ‘SA Demons’ affecting driving performance
conversing with a passenger, cell phone conversations
include increased cognitive workload, attention
may differ in terms of effects on SA and performance
tunneling, misplaced salience, and data overload,
since the caller cannot visualize the driving situation to
among others (for a detailed discussion on ‘SA Demons,’
assist the driver, and the driver may need to use one
see [34]). Part of the reason for identifying these is to
hand to hold the telephone [4].
illustrate that the negative effects of these problems can
be minimized through careful system design and
While this research on driving performance suggests the
training. For example, to prevent technology-induced
possibility of decrements in SA at multiple levels for
performance problems [35] in lower level (operational
some visual-manual devices used during driving, it has
and tactical) driving behaviors, the reliability and manner
also been hypothesized that conversations alone may
of delivery of information is critical in the design of in-
compromise drivers’ comprehension of stimuli perceived
vehicle navigation systems. This has already been
from the driving environment relative to the goal of
illustrated through the discussion of the Travtek findings
maintaining vehicle safety, specifically sudden stopping
– which served to demonstrate that systems can be
situations. The notion underlying these hypotheses is
designed such that information is presented reliably,
that cell phone conversations when driving may compete
clearly, and meaningfully to drivers without distracting
for limited driver cognitive resources, possibly leading to
their attention away from the main driving task (operating
SA decrements and decreases in driving performance.
the vehicle) or increasing the cognitive workload
associated with understanding the information provided
Interestingly, again the empirical record provides
by the system. In addition, driver safety programs can be
conflicting data. In one study, speech-based interactions
developed to increase awareness of the dangers and
with in-vehicle computers were observed to have poorly
risks associated with device usage during driving. As will
designed interfaces that may distract drivers and
be discussed next, to evaluate the effectiveness of
increase reaction time [30], effects similar to some cell
proposed technology design concepts and training
phone interfaces. Other studies, which have used both
interventions, researchers must be able to validly and
behavioral and brain imaging measures to examine the
reliably measure SA.
effects of conversation on reaction time to events in a
driving-like environment reveal that conversations have
a negligible effect on rates of missed detections (which
MEASURING SITUATION AWARENESS
were not distinguishable from “just driving/no-
Although we can build vehicles and equipment to
conversation” epochs) and resulted in reliable but small
support SA and even develop intervention strategies
increases in reaction times to events (<~100 ms) [31,
32]. When these differences for conversation in the such as training programs to improve SA, understanding
laboratory are compared with the effects that similar what factors influence driver SA and how and where SA
impacts driving performance is a critical requirement for
research and design. As one example of how important
this is, when new, back-up cameras and related backing
crash countermeasure systems were introduced,
extensive research was undertaken to study how drivers
behaved during backing. These studies examined a
range of SA-related topics, from where drivers looked
during backing, to what they attended to and noticed, to
what they were aware of, and how they responded to
unexpected obstacles placed out-of-view in the backing
path. See [36] and [37] as two examples of the kind of
studies that are important in the early development of
new technologies, particularly those that may affect SA.
Further, it is important to recognize that a range of
fundamental studies are required to empirically
determine the impact of any particular countermeasure
system on driver SA in a driving task, as well as to
identify any unintended consequences.
Therefore, in this section, the focus is not on whether to
measure SA – but how. Applying the model of SA
discussed earlier, researchers can determine if a new
device or modification to an existing one is helping or
hindering the driver’s ability to perceive critical
information (Level 1 SA), comprehend the relevance of
this information to the driving task (Level 2), and use this
information to predict what will happen next (Level 3 SA)
as well as to evaluate how these effects on driver SA
influence decision making and, ultimately, driving safety
and performance. Thus, measurement of SA provides a
degree of diagnosticity that is useful in training and
system design and evaluation. SA measurement can be
used to identify the source of the problem as well as
establish a baseline for comparison of the effects of
different training or design solutions.
In general, methodologies vary in terms of direct
measurement of SA (e.g., objective real-time probes or
subjective questionnaires assessing perceived SA) or
indirect methods (e.g., process indices) that infer SA
based on operator behavior or performance. Direct
measures are typically considered to be ‘product-
oriented’ in that these techniques assess an SA
outcome; indirect measures are considered to be
‘process-oriented,’ focusing on the underlying processes
or mechanisms required to achieve SA [38]. The
selection of measurement choice depends upon what
the researcher or developer wants to know about SA and
what facilities or setup is available to implement the
measures. Examples of each of these SA measurement
approaches will be further described next.
PROCESS INDICES - Simply stated, process indices,
such as psycho-physiological measures, examine how
individuals process information in their environment [39].
Such measures include PET, MRI, EEG, eye blinks, eye
tracking and cardiac activity, just to name a few.
Tracking eye movements is one of the more common
psycho-physiological approaches for providing insight
into perception and comprehension. In the context of
assessing driver SA, eye tracking devices can be used
to monitor where the driver is directing his/her attention.
More specifically, is the saliency of important cues
sufficient or are nonessential cues drawing away the
driver’s attention? Is the driver’s gaze focused more on
the external environment or on the interface of an in-
vehicle device (e.g., advanced information display) or on
the rear view mirror (a display to support SA)? Analyzing
communications can also serve as process indices of
driver SA. For example, verbalizations made by drivers
during a navigation task can be analyzed to determine
how well information is being acquired from an in-vehicle
navigation system.
Process indices are advantageous in that these offer
objective assessment of driver SA and provide an
indication of information access and utilization. However,
process indices create large amounts of data to analyze
and are difficult to implement in the real world
environment (e.g., eye tracking devices, if head-
mounted, can be cumbersome and intrusive). Further,
process indices do not indicate what is actually done
with the information acquired (processing) nor whether
the information is registered correctly or what is retained
in memory. Instead, these measures simply indicate that
the driver looked at the information. Given these
limitations, process indices are more suitable for
investigating specific research questions of information
acquisition and for examining the processes underlying
SA (e.g., perception, attention) rather than the final
product.
Studies using process indices have yielded mixed
results, in part, because it is so difficult to tease out what
is SA and what are indicators of perception, information
processing, and workload. In one of the earliest studies
used to examine SA and process indices, Poe,
Suyenobu, Bolstad, Endsley, and Sterman [40] did not
find any correlation between EEG activity and SA as
measured by the SAGAT methodology (described next).
Results indicated that individual differences played a
large role in correlating between EEG metrics and
workload and performance. However, this study used
technology that is now considered outdated and newer
measurement systems may yield different results.
Results of a study of combat pilots conducted by
Vidulich, Stratton, Crabtree, and Wilson [41] revealed
that EEG and eye blinks were related to the
maintenance of SA. Essentially, the metrics were able to
detect when pilots had to work harder (i.e., experienced
higher workload) to maintain their SA.
STATE-OF-KNOWLEDGE METRICS – State-of-
knowledge metrics include both objective and subjective
direct measures of SA. Objective measures directly
assess SA by comparing an individual’s perceptions of
the situation or environment to some ‘ground truth’
reality. Specifically, objective measures can be used to
collect data from drivers’ perceptions of the situation and
compare this to what is actually happening at a given
moment in time. Thus, this type of assessment provides
a direct measure of SA and does not require drivers or
experimenters to make judgments about situational
knowledge on the basis of incomplete information.
Objective measures can be gathered in one of three
ways: during an interruption in task performance (e.g.,
SAGAT), real time as the task is completed (e.g.,
probes), or post-test following completion of the task.
SAGAT Queries - One common approach to directly and
objectively measure SA is the Situation Awareness
Global Assessment Technique (SAGAT) [42]. SAGAT
utilizes a concurrent memory probe technique that
presents queries related to the current task environment.
The SAGAT queries are derived from a goal directed
task analysis (GDTA) of the domain being trained or
evaluated during the exercise. A GDTA is a unique form
of cognitive task analysis that involves conducting
extensive knowledge elicitation sessions with domain
subject matter experts (for a detailed description of this
methodology; see [34]). The objective of the GDTA is to
identify the major goals, subgoals, and decisions that
drive performance in a particular job or position as well
as to delineate the critical, dynamic information
requirements associated with each goal and decision. To
illustrate this methodology, consider that an important
subgoal in driving is to “Avoid road hazards.” A key
decision for this subgoal is “What type of hazard is
present?” The associated SA information requirements
for this decision are listed in Table 1. Note that this
GDTA example only illustrates one subgoal for the
driving task. The actual GDTA for this domain is much
more comprehensive and detailed, involving multiple
subgoals, decisions, and associated SA information
requirements (see [43]).
Administration of the SAGAT involves freezing a
simulation exercise at randomly selected times and
hiding task information sources (e.g., blanking visual
displays) while individuals quickly answer randomly
Table 1: Example of GDTA for the Driving Task Listing SA Information Requirements for the Subgoal “Avoid Road
Hazards” (adapted from [43]).
Subgoal: Avoid Road Hazards
Decision: What type of hazard is present?
SA Information Requirements: Assess the type of hazards
• Pedestrians
• Crosswalks
• Signs present
• School zones
• Signs
• Presence of children
• Construction zones
• Signs present
• Speed limit changes
ordered questions about their current perceptions of the
situation. These responses are then compared to
‘ground truth’ (i.e., actual data on the real situation) to
assess the accuracy of the individuals’ SA. SAGAT
offers several advantages including: requires no post
data collection, provides a direct objective measure of
SA minimizing potential biasing of operator SA, utilizes
random sampling of queries to provide an unbiased
measure of SA, and has been extensively validated in
numerous complex domains [42]. SAGAT, however,
does have its limitations. The temporary halt in the
exercise may be perceived as disruptive to task
performance and may also potentially increase cognitive
workload. In addition, because it involves ‘freezing’ task
performance, SAGAT is better suited for assessing SA in
simulation exercises and may not be practical for real-
time measurement of SA. Finally, while the SAGAT
methodology is relatively straightforward, the greatest
challenge is creating the appropriate queries. Generation
of the queries requires conducting a detailed task
analysis to identify domain-relevant SA information
requirements. Care should be taken to ensure the
queries address all levels of SA and that these can be
objectively scored.
Bolstad [44] used the SAGAT methodology to assess
age-related changes associated with driving and how
this affects SA formation. She found that SA did decline
with age, but like other performance metrics, the
variability increased significantly. Additionally, the
SAGAT methodology provides a degree of diagnosticity
not found with the other measures. For example, Bolstad
and Endsley [45] were asked to assess the ability of a
new military force structure to support data fusion and
analysis. Using the model of SA cited earlier, Levels 2
and 3 SA best represent this ability. The SAGAT
methodology was used to measure SA of Army officers
while performing a command and control task. Results
showed that SA scores were higher for Levels 2 and 3
SA than for Level 1 SA, thus, supporting the study’s
hypothesis.
• Road conditions
• Detours
• Signs present
• Car direction of travel
• Curves
• Signs present
• Car direction of travel
• Speed limit changes
• Unsafe roads
 • Road lane changes (2 to 1)
• Presence of road equipment
Real-time probes - Real-time probes (open questions
embedded as verbal communications during the task)
can be utilized to naturally and unobtrusively measure
driver SA during task performance [46]. Real-time
probes are similar to SAGAT in that these can be used
to query drivers on their knowledge of key task-relevant
information in the environment; however, this approach
offers an important advantage over the SAGAT in that
task performance is not disrupted (i.e., the simulation or
task is not stopped) but rather the queries are
incorporated as a natural part of the task. Thus, real-time
probes are more ideally suited for assessing SA in the
real world environment. In addition, driver SA can be
evaluated both in terms of response times as well as
accuracy. As with the SAGAT, however, real-time probes
do have some limitations. The probes must be created
using a cognitive task analysis in order to specifically
assess SA requirements. Further, if not seamlessly
integrated within the task, probes may interfere with
driver performance. Because of their specificity and link
to the current situation, probes also limit the amount of
information that can be collected. Thus, administration of
real-time probes requires careful synchronization with
tasks or events and should only be presented one
question at a time. Finally, multiple measures are
needed to ensure reliability.
In an investigation of the sensitivity and validity of real-
time probes, Jones and Endsley [46] compared the use
of questions directly posed to operators of an air defense
system during simulated peace and wartime operations
with the SAGAT. The probes used in this study were
based on an SA requirements analysis for the air
defense system and were verbally administered one-at-a
time during task performance. Results showed the SA
probes to be sensitive to workload manipulations in
terms of overall response time and accuracy of
responses. (Response time was measured because all
information displays remained visible to participants
during probes.) With respect to the validity of the real-
time probes for assessing SA, results indicated
significant, but weak, correlations of response accuracy
with the accuracy of responses to the SAGAT queries,
which were also found to be a sensitive measure of SA.
Additionally, real-time probes showed differences
between scenarios, but not by query while SAGAT
showed differences at the individual query level, thereby
providing diagnostic detail as to the types of SA
differences that existed between the two scenarios.
Jones and Endsley recommended that at least 6 to 8
repeats of each probe type be used in order to obtain
sufficient reliability and sensitivity in assessing
participant cognitive states. In their experiment, a probe
• Road condition
• Signs present
• Behavior of other drivers
occurred, on average, once every 2 minutes and did not
appear to interfere with performance, suggesting that
probes could be administered even more often without
posing problems.
Post-test questionnaires and post-test analysis - As the
name implies, post-test questionnaires are administered
following task completion, and, as such, do not interfere
with driving performance and, thus, are less intrusive. In
addition, because the driver has completed the task and
time is less of an issue, researchers can administer a
more detailed yet still objective assessment of SA. This
methodology is relatively easy to implement in either
simulated or real world environments. A major drawback
to this approach is the risk of rationalization or
generalization after the fact, that is, responses are based
on the driver’s memory of what occurred in the past
rather then his/her current understanding of the situation
at that given moment in time. Accordingly, post-test
questionnaires are limited to providing reliable
assessment of SA only with regard to the end of the task
or event. Still, this methodology can be useful for
determining the overall success of a new technology.
Subjective measures (described next) are commonly
used to assess SA on a post-test basis. Additionally, it is
not uncommon to utilize SAGAT at the end of a scenario
or test. However, SA in our work has been shown to vary
throughout an experiment and assessing it only at the
end does not provide the researcher with a clear picture
of what is actually occurring. Typically, we find that SA
improves over time in our longer experimental studies
(e.g., [47]).
Post-test analysis can be performed on existing sets of
data to determine their causal factors. For example in
the driving domain, detailed databases on crash data
can be examined to determine which SA errors led to the
specified incidents. The greatest drawback to this
approach is the fact that the analysis is being performed
in retrospect. Thus, the analysis can only be performed
at a very broad level and the actual series of events that
led up the crash may never be known.
A number of studies have been performed that employ
this methodology. For example, an analysis of SA errors
in aviation was conducted by Jones and Endsley [48]
using reports from the NASA Aviation Safety Reporting
System (ASRS). They used an SA error taxonomy based
on Endsley’s [2] model of SA. The aviation errors cited in
this database were primarily due to Level 1 SA problems
(76.3%) including pilot’s failure to monitor or observe the
needed data. Level 2 errors occurred only 20.3% of the
time. Rodgers, Mogford and Strauch [49] also performed
a post-hoc assessment of SA with regard to Air Traffic
Controller incidents. They initially determined that loss of
awareness was the primary reason for the events, but
further analyses using Endsley’s model of SA allowed for
a more detailed description. Three causal factors were
then used to reassess the loss of awareness incidents:
failure to detect displayed information, failure to
comprehend displayed information, and failure to project
the future status of displayed information.
Subjective measures - Subjective measures directly
assess SA by asking individuals (or experienced
observers) to rate their SA on an anchored scale (for a
detailed review, see [50]). These ratings can be
collected during task performance or following task
completion. Subjective measures of SA are attractive in
that these are relatively straightforward, inexpensive,
and easy to administer. However, several important
limitations should be noted. Individuals making
subjective assessments of their own SA are often
unaware of information they do not know. Further, self-
ratings may be tainted by performance outcomes.
Subjective measures also tend to be global in nature,
and, as such, do not fully exploit the multivariate nature
of SA to provide the detailed diagnostics available with
objective measures. Nevertheless, self-ratings may be
useful in that these can provide an assessment of
drivers’ degree of confidence in their SA.
Subjective estimates of an individual’s SA may also be
made by experienced observers (e.g., supervisors or
trained external experts). These observer ratings may be
somewhat superior to self-ratings of SA because more
information about the true state of the environment is
usually available to the observer than to the driver, who
may be focused on performing the driving task (i.e.,
trained observers may have more complete knowledge
of the situation). However, observers have only limited
knowledge about the driver’s concept of the situation and
cannot have complete insight into the mental state of the
individual being evaluated. Thus, observers are forced to
rely more on drivers’ observable actions and
verbalizations in order to infer their level of SA. Given the
aforementioned limitations, subjective measures,
whether self or observer ratings, should not be used in
isolation, but rather to complement objective techniques.
An example of a domain-specific subjective measure of
SA is the Mission Awareness Rating Survey (MARS),
which consists of two 4-item subscales [51]. The SA
Content subscale requires participants to rate how well
they can identify, comprehend, and predict (the three
levels of SA as defined by Endsley [2]) as well as decide
in the given mission. The SA Workload subscale
requires participants to rate their difficulty, in terms of
how much mental effort is required, to identify,
comprehend, predict, and decide in the given mission.
Responses are recorded using a four-point anchored
scale, with different anchors for each item. The utility of
MARS as a subjective measure of SA was evaluated
during an experiment designed to assess four different
approaches to simulating night vision goggles in a virtual
environment [51]. Sixteen enlisted soldiers, working in
four-man teams, completed four MOUT (Military
Operations in Urban Terrain) missions in a virtual
environment. After each mission, participants completed
a MARS questionnaire. The results indicated that MARS
significantly and robustly differentiated among the four
means of simulating night vision goggles, and that the
MARS data were consistent with other objective and
subjective indexes designed to assess the four different
approaches to night simulations.
PERFORMANCE AND BEHAVIORAL-BASED
MEASURES - Performance measures ‘infer’ SA from the
end result (i.e., task performance outcomes), based on
the assumption that better performance indicates better
SA. Common performance metrics include quantity of
output or productivity level, time to perform the task or
respond to an event, and the accuracy of the response
or, conversely, the number of errors committed. The
main advantage of performance measures is that these
can be collected objectively and without disrupting task
performance. However, although evidence exists to
suggest a positive relation between SA and
performance, this connection is probabilistic and not
always direct and unequivocal [2]. In other words, good
SA does not always lead to good performance and poor
SA does not always lead to poor performance [52].
Behavioral measures also ‘infer’ SA from the actions that
individuals choose to take, based on the assumption that
good actions will follow from good SA and vice-versa.
Behavioral measures rely primarily on observer ratings,
and, thus, are somewhat subjective in nature. To
address this limitation, observers can be asked to
evaluate the degree to which individuals are carrying out
actions and exhibiting behaviors that would be expected
to promote the achievement of higher levels of SA. This
approach removes some of the subjectivity associated
with making judgments about an individual’s internal
state of knowledge by allowing them to make judgments
about SA indicators that are more readily observable.
Within the context of driving, observers can be asked to
evaluate observable behaviors such as, for example,
whether or not the driver brakes in time to avoid a
collision, notices a ball rolling across the street,
maintains a consistent speed or following distance, or
responds to an inaccurate display setting.
A significant limitation to the utility of performance and
behavioral-based measures is that these methodologies
‘infer’ rather than directly assess SA and thus, lack
sensitivity and diagnosticity. This also makes the
interpretability of the data collected from these measures
problematic and subjective. Thus, performance and
behavioral-based measures should be used in
conjunction with others measures of SA that directly
assess this construct. It is also recommended that these
measures cover a wide range of tasks or scenarios to
increase generalizability.
The Situation Awareness Behaviorally Anchored Rating
Scale (SABARS) is a behavioral measurement approach
consisting of 27 items, derived from an SA requirements
analysis, that identify key observable behaviors related
to developing SA in a MOUT environment, such as
communicating clearly with higher echelons,
subordinates, and adjacent units [53]. One additional
item concerns the platoon leader’s overall SA. Expert
trained observers rate participants on the SA-related
behaviors assessed by the 28 items using a 5-point
scale, with performance on specified behaviors rated as
very poor, poor, borderline, good, or very good (a “not
applicable” option is also available). Strater et al. [53]
conducted a comparison of SA measures to determine if
the SABARS was predictive of the level of SA measured
using the SAGAT. Observations on individual SAGAT
queries were used as the response measure in a
regression model in order to establish whether any of the
SABARS items were significantly related to SAGAT
performance. Results showed that some of the SABARS
factors accounted for significant portions of variance in
some SAGAT questions. Results of the analysis also
showed the SABARS to be sensitive to the experience
levels of soldiers.
MODELING SA - Rather than simply measure SA, the
goal of SA modeling approaches is to objectively predict
SA based on readily observable verbal and non-verbal
communications. Specifically, team communications
(particularly verbal communications) support the
knowledge building and information processing that
leads to SA construction [54]. Thus, since SA may be
distributed via communication, computational linguistics
and machine learning techniques can be combined with
natural language analytical techniques (e.g., Latent
Semantic Analysis) to create models that draw on the
verbal expressions of the team to predict SA and task
performance [55, 56]. For example, the Automated
Communication and Situation Awareness (ACASA) tool
offers real-time, non-intrusive, quantitative assessment
of SA by analyzing communication exchanges among
team members [57]. Since the communication data is
collected using either automatic speech recognition
(ASR) software or transcriptions of speech recordings,
this methodology does not interrupt activities or affect
task performance. Thus, SA modeling approaches, such
as the ACASA tool, are appropriate for use in both
simulations and real world environments. Further, this
methodology can provide diagnostic information
regarding current SA. For example, when coupled with
ASR software, the ACASA tool can be used to quickly
identify if immediate action needs to be taken to address
poor SA among team members.
Although evidence exists to support the utility of
communication analysis for predicting team SA [57], time
constraints and technological limitations (e.g., cost and
availability of speech recording systems and speech-to-
text translation software) may make this approach less
practical and viable in time-pressured, fast paced
operations. In addition, the models generated using this
approach are domain and task specific; thus, unique
models must be created for each environment or
application. Lastly, this measure is only effective for
measuring SA in a team environment and would not be
suitable for situations in which a single operator is being
evaluated.
SUMMARY - Improving driver performance requires
carefully evaluating proposed design options for the
displays and interfaces of in-vehicle devices and
automation technologies. The measurement of driver SA
provides critical information with greater sensitivity than
simple performance measures. However, the choice of
which measure(s) to use is important. Considerations
must be given to each measurement approach in terms
of the following criteria:
• Validity - Is it measuring what it is intended to
measure?
• Reliability - Does the measure remain consistent?
With what else does it correlate?
• Sensitivity - Will the measure detect differences in
SA? What is a good SA score?
• Operational Constraints – What are the task’s
dynamic priorities? Where will measurement be
located? What is the cost to implement? How much
time is required to analyze the data?
In general, objective measures of SA are the most
sensitive and diagnostic. Subjective measures of SA are
more related to confidence in one’s SA abilities and
performance measures require SA to be inferred, rather
than actually providing direct measures of SA.
In addition, the choice of which measurement approach
to utilize greatly depends on the research question that
is being addressed. Process indices (e.g., eye tracking)
can be used to examine the ability of drivers to look at
the information needed for SA development. Direct,
objective state-of-knowledge metrics (e.g., SAGAT, real-
time probes) are useful for assessing continuous levels
of SA during driving; for example, SAGAT has been
used successfully to examine age-related differences in
driving performance [44]. Alternately, direct, subjective
measures (e.g., MARS) are more suitable for soliciting
self-reports of drivers’ confidence in their SA abilities.
Performance and behavioral-based measures may be
preferred when the focus is on drivers’ observable
behaviors or a desired outcome. Finally, SA modeling
approaches are especially useful for assessing the SA
among two or more team members.
CONCLUSION
The measurement of situation awareness has great
potential for use by scientists and automotive system
engineers to assess and design systems for drivers and
their passengers. Understanding how SA changes with
the addition of a new system or how SA varies among
different designs and how this is linked to performance
provides the researcher with a very valuable tool. The
diagnosticity of SA measurement is of particular
importance. Knowing that drivers feel they have better
SA with a new device is just as important as measuring
a real improvement in task performance. This is
especially critical in today’s market as science and
technology are changing at an extraordinarily fast pace.
In particular, the use of in-vehicle devices (both
integrated with the vehicle and carried-in) and advanced
automation technologies is expanding. These devices
represent a critical mediating factor in linkages of SA to
specific driving behaviors and performance, with the
potential for both positive and negative effects [4]. Future
automotive environments pose interesting and unique
challenges for SA as drivers are tempted with many new
technologies that will vie for their attention and can
degrade awareness of road conditions and other drivers.
Conversely, these same technologies offer potential
improvements in SA over the present day by making
some tasks easier or by augmenting human sensory
capacity. Accordingly, the development of successful
and appropriate new technologies for the automobile is
dependent on consideration of factors that impact driver
SA as well as careful measurement of the both the
positive and negative effects of these factors.
Furthermore, researchers must consider if these new
technologies improve performance across the entire age
range of product users – the drivers. Unlike military
applications, in which the population is relatively small
and homogeneous, automobile drivers and passengers
are quite varied and the age range can be very broad.
Research has shown that older adults have less SA than
younger adults [44], and that older adults usually perform
less well with new technology [58]. Thus, the integration
of new technology into the automobile should take this
research issue into consideration and measure SA and
performance to ensure that a system which improves SA
for younger drivers does not impede SA for older drivers.
Researchers also need to be cognizant that many
metrics of performance and workload, which are thought
to measure SA, are in fact measuring something
different. This is not to say that these are not useful
metrics, but rather they are assessing an inter-related
construct and thus, should be used in combination with a
validated SA metric to provide a more diagnostic
interpretation of the results. For example, several of the
authors in this book (see chapters by Angell and van der
Horst) have utilized a secondary detection task as a
means of assessing workload. They state that when
performance on a secondary task declines due to high
workload, the assumption is that SA declines as well.
While this is a valid hypothesis, it may not always be
true. While it is possible that high workload will be
associated with low SA, it also possible to have high
workload and still have high SA. Indeed, several studies
that compared SA and workload metrics revealed a
disassociation between these two attributes [59, 60].
Endsley [59] states that the relationship between SA and
workload are “flip sides of the same coin.” Essentially,
SA is a precursor to performance and workload is
expended to achieve and maintain SA, but the
relationship is not always clear. In general, one would
expect that a reduction in SA would be associated with a
reduction in driving performance — a direct
correspondence. This is not strictly so, however. The
loss of SA merely puts the driver at an increased risk of
a performance error. Prior studies have shown that good
SA is necessary, but not sufficient for good performance
[52, 61]. Thus, metrics that address workload and
performance should be coupled with valid SA metrics to
determine how best to support SA while lowering the
workload of the driver.
Finally, one important issue that still remains is to
determine what actually constitutes good SA. The
measurement of SA provides a powerful tool to
researchers and vehicle system designers. However,
what is considered a good level of SA for the driving task
has not yet been established. For example, if for a
sample of drivers, SA and performance is being
measured while the drivers each are using a vehicle
equipped with a new device and the drivers have fairly
high performance scores, high subjective SA scores, but
only moderate objective SA (SAGAT) scores (~50%), is
the new device considered a success? Since SAGAT is
highly dependent upon the queries being asked, it may
be that the queries selected are too difficult for the
drivers to answer using the available systems.
Additionally, since subjective SA is high and
performance is high, objective SA may not be
considered as essential. Thus, researchers must
consider the hypotheses for their experiments and adopt
the criterion that best fits their research questions. In
sum, there is no clear right or wrong approach to
measuring SA as, ultimately, the most appropriate level
of SA is context-specific, varying from person to person
and situation to situation.
Nonetheless, in looking at the literature on the use of
surrogate methods to assess the effects of new devices
and tasks on driver performance and behavior, it would
appear that the theoretical framework of SA, as well as
the practical measurement tools that are available within
its umbrella, could complement the research already
underway in this area. A review of the literature suggests
that a great deal of research and measurement has
been directed to what would be characterized as
assessing general SA, while more validated metrics
provide an assessment of the levels of SA. This will
provide more diagnostic feedback in terms of how
drivers are reacting and what can be done to improve
their performance. Perhaps it is in this domain that
situation awareness methods and theory will offer new
insights as future research is initiated.
ACKNOWLEDGMENTS
Portions of this paper were presented at an invited
lecture at the SAE Driver Metrics Workshop, San
Antonio, TX, 2-3 June 2008. The authors wish to
express appreciation to Mr. Anthony M. Costello for his
assistance in finalizing this chapter. The authors also
wish to thank the reviewers for their comments which
greatly enhanced the contribution of this chapter.
REFERENCES
1. Endsley, M.R. (1997). The Role of Situation
Awareness in Naturalistic Decision Making. In C.E.
Zsambok & G. Klein (Eds.), Naturalistic Decision
Making, 269-283. Mahwah, NJ: LEA.
2. Endsley, M.R. (1995). Toward a Theory of Situation
Awareness in Dynamic Systems. Human Factors
37(1), 32-64.
3. Endsley, M.R. & Kiris, E.O. (1995). The Out-of-the-
Loop Performance Problem and Level of Control in
Automation. Human Factors, 37(2), 381-394.
4. Ma, R., & Kaber, D.B. (2005). Situation Awareness
and Workload in Driving while Using Adaptive Cruise
Control and a Cell Phone. International Journal of
Industrial Ergonomics, 35, 939-953.
5. Adams, M., Tenney, Y., & Pew, R. (1995). Situation
Awareness and the Cognitive Management of
Complex Systems. Human Factors, 31(1), 85-104.
6. Gaba, D.M., Howard, S.K., & Small, S.D. (1995).
Situation Awareness in Anesthesiology. Human
Factors, 37, 20-31.
7. Salas, E., Prince, C., Baker, D.P., & Shrestha, L.
(1995). Situation Awareness in Team Performance:
Implications for Measurement and Training. Human
Factors, 37(1), 123-136.
8. Endsley, M.R. (1988). Situation Awareness Global
Assessment Technique (SAGAT). Proceedings of
the National Aerospace and Electronics Conference
(NAECON), 789-795.
9. Endsley, M.R. (1993). Situation Awareness: The
Development and Application of a Theoretical
Framework. Proceedings of the Human Factors
Society 37th Annual Meeting, 39-40. Santa Monica,
CA: Human Factors and Ergonomics Society.
10. Ward, N.J. (2000). Automation of Task Processed:
an Example of Intelligent Transportation Systems.
Human Factors and Ergonomics in Manufacturing,
10(4), 395-408.
11. Matthews, M.L., Bryant, D.J., Webb, R.D., &
Harbluk, J.L. (2001). Model of Situation Awareness
and Driving. Transportation Research Record, 1779,
26-32.
12. Gugerty, L.J., & Tirre, W.C. (2000). Individual
Differences in Situation Awareness. In M.R. Endsley,
& D.J. Garland (Eds.), Situation Awareness Analysis
and Measurement, 249-276. Mahwah, NJ: Erlbaum.
13. Gugerty, L.J. (1997). Situation Awareness during
Driving: Explicit and Implicit Knowledge in dynamic
spatial memory. Journal of Experimental
Psychology: Applied, 3(1), 42-66.
14. Treat, J.R., Tumbas, N.S., McDonald, S.T., Shinar,
D., Hume, R.D., Mayer, R.E., Stansifer, R. L.,
&Catellan, N.J. (1979). Tri-level Study of the Causes
of Traffic Accidents: Final Report Volume I: Causal
Factor Tabulations and Assessments. Institute for
Research in Public Safety (DOT Publication No.
DOT HS-805 085). Bloomington, IN: Indiana
University.
15. Shinar, D. (1978). Psychology on the Road: The
Human Factor in Traffic Safety. New York: Wiley.
16. Sheridan, T.B. (2002). Human and Automation:
System Design and Research Issues. New York:
Wiley.
17. Kaber, D.B. & Endsley, M.R. (2003). The Effects of
Level of Automation and Adaptive Automation on
Human Performance, Situation Awareness and
Workload in a Dynamic Control Task. Theoretical
Issues in Ergonomic Science, 5(2), 113-153.
18. Rudin-Brown, C.M., Parker, H.A., & Malisia, A.R.
(2003). Behavioral Adaptation to Adaptive Cruise
Control. Proceedings of the 47th Annual Meeting of
the Human Factors and Ergonomics Society. Santa
Monica, CA: Human Factors and Ergonomics
Society.
19. Kiefer, R. & Salinger, J. (2006). Automotive Collision
Avoidance System Field Operational Test Program.
Presented at the US DOT Public Meeting on the
Benefits of Advanced Crash Avoidance Systems.
Ypsilanti, Michigan.
http://www.itsa.org/itsa/files/pdf/ACAS GM_UMTRI
FINAL.pdf.
20. Michon, J.A. (1985). A critical view of driver behavior
models: What do we know, what should we do? In L.
Evans & R. C. Schwing (Eds.), Human Behavior and
Traffic Safety. New York: Plenum Press.
21. Ma, R., & Kaber, D.B. (2007). Situation Awareness
and Driving Performance in a Simulated Navigation
Task. Ergonomics, 50(8), 1351-1364.
22. Dingus, T., McGehee, D., Hulse, M., Jahns, S.,
Mannakkul, N., Mollenhauer, M., &Fleischman, R.
(1995) Travtek Evaluation Task C3 – Camera Car
Study. FHWA-RD-94-076. McLean, VA: Federal
Highway Administration.
http://www.itsdocs.fhwa.dot.gov/JPODOCS/REPTS_
TE/5303.pdf.
23. Redelmeier, D.A., & Tibshirani, R.J. (1997).
Association Between Cellular-telephone Calls and
Motor Vehicle Collisions. New England Journal of
Medicine, 336(7), 453-458.
24. Violanti, J.M. (1997). Cellular Phones and Traffic
Accidents. Public Health, 111(6), 423-428.
25. Young, R.A. (2001).Association between embedded
cellular phone calls and vehicle crashes involving
airbag deployment. Proceedings of Driving
Assessment 2001: The First International Driving
Symposium on Human Factors in Driver
Assessment, Training and Vehicle Design, 1, pp.
390–400. http://ppc.uiowa.edu/Driving-
Assessment/2001/Summaries/Driving%20Assessme
nt%20Papers/81_young-richard.htm
26. Young, R.A. & Schreiner, C.S. (2008), “Personal
conversations on a hands-free embedded wireless
device while driving: 1.Real-world airbag deployment
crash risk,” Risk Analysis. DOI: 10.1111/j.1539-
6924.2008.01146.x.
http://www3.interscience.wiley.com/journal/1214949
51/abstract
27. Hahn, R. W. & Prieger, J. E. (2006). The Impact of
Driver Cell Phone Use On Accidents. Advances in
Economic Analysis & Policy, Volume 6, Issue 1,
Article 9.
28. Hancock, P.A., Lesch, M., & Simmons, L. (2002).
The Distraction Effects of Phone use During a
Crucial Driving Maneuver. Accident Analysis and
Prevention, 1-14.
29. Hancock, P.A., Simmons, L., Hashemi, L., Howarth,
H., & Ranney, T. (1999). The Effects of In-Vehicle
Distraction on Driver Response During a Critical
Driving Maneuver. Transportation Human Factors,
1(4), 295-309.
30. Lee, J.D., Caven, B., Haake, S., & Brown, T.L.
(2000). Speech-Based Interaction with In-Vehicle
Computers: The effect of Speech-Based E-mail on
Drivers’ Attention to the Roadway, presented at the
Driver Distraction Internet Forum. Retrieved
December 22, 2008, from
http://www.engineering.uiowa.edu/~csl/publications/i
ndex.html#00.
31. Bowyer, S.M., Hsieh, L., Moran, J.E., Young, R.A.,
Manoharan, A., Liao, C.-C., Malladi, K., Yu, Y.-J.,
Chiang, Y.-R., & Tepley, N. (2009). Conversation
effects on neural mechanisms underlying reaction
time to visual events while viewing a driving scene
using MEG. Brain Research, 1251, 151-161.
http://dx.doi.org/10.1016/j.brainres.2008.10.001
32. Hsieh, L., Young, R., Bowyer, S.M., Moran, J.E.,
Genik, R.J. II, Green, C.C., Chiang, Y.R., Yu, Y.J.,
Liao, C.C., &Seaman, S. (2009). Conversation
effects on neural mechanisms underlying reaction
time to visual events while viewing a driving scene:
fMRI analysis and asynchrony model. Brain
Research 1251, 162–175.
http://dx.doi.org/10.1016/j.brainres.2008.10.002
33. Angell, L., Auflick, J., Austria, P. A., Kocchar, D.,
Tijerina, L., Biever, W., Diptiman, T., Hogsett, J. &
Kiger, S. (2006). Driver Workload Metrics Task 2
Final Report & Appendices. Final Research Report:
DOT HS 810 635. NHTSA/USDOT, Washington,
D.C. Available at http://www-
nrd.nhtsa.dot.gov/pdf/nrd-
2/Driver%20Workload%20Metrics%Final%20Report.
pdf&http://www.nhtsa.dot.gov/staticfiles/DOT/NHTS
A/NRD/Multimedia/PDFs/Crash%20Avoidance/Drive
r%20Distraction/Driver%20Workload%20Metrics_ap
pendices.pdf.
34. Endsley, M.R., Bolte, B. & Jones, D.G. (2003).
Designing for Situation Awareness: An Approach to
User-Centered Design. New York: Taylor & Francis.
35. Parasuraman, P.R. & Riley, V. (1997). Humans and
Automation: Use, Misuse, Disuse, Abuse. Human
Factors, 39(2), 230-253.
36. Llaneras, R. E., Green, C.A., Kiefer, R. J., Chundrlik,
W., Altan, O.D., & Singer, J. P. (2005). Design and
Evaluation of a Prototype Rear obstacle Detection
and Driver Warning System, Human Factors, 47(1),
199-215.
37. McLaughlin, S.B., Hankey, J. M., Green, C. A., &
Kiefer, R. J. (2003). Driver Performance Evaluation
th
of Two Rear Parking Aids. Proceedings of the 18
Enhanced Safety Vehicle Conference.
38. Graham, S.E. & Matthews, M.D, (2000). Modeling
and Measuring Situation Awareness. In J.H. Hiller &
R.L. Wampler (Eds.), Workshop on Assessing and
Measuring Training Performance Effectiveness
(Tech. Rep. 1116) (pp. 14-24). Alexandria, VA: U.S.
Army Research Institute for the Behavioral and
Social Sciences.
39. Wilson, G.F. (2000). Strategies for
Psychophysiological Assessment of Situation
Awareness. In M.R. Endsley & D.J. Garland (Eds.),
Situation Awareness Analysis and Measurement,
175-188, Mahwah, NJ: Lawrence Erlbaum
Associates.
40. Poe, G.R., Suyenobu, B.Y. et al. (1991). EEG
Correlated of Critical Decision Making in Computer
Simulated Combat. Sixth International Symposium
on Aviation Psychology, Columbus, OH, Department
of Aviation, The Ohio State University.
41. Vidulich, M.A., Stratton, M., Crabtree, M., & Wilson
G. (1994). Performance-Based and Physiological
Measures of Situational Awareness. Aviation, Space
and Environmental Medicine, 65 (5 Suppl.), A7-A12.
42. Endsley, M.R. (1995). Measurement of Situation
Awareness in Dynamic Systems. Human Factors,
37(1), 65-84.
43. Bolstad, C.A. (2001) Age-Related Factors Affecting
the Perception of Essential Information during
Complex Driving Situations. Unpublished
Dissertation, North Carolina State University,
Raleigh.
44. Bolstad, C.A. (2001). Situation Awareness: Does It
Change with Age? Proceedings of the Human
Factors and Ergonomics Society 45th Annual
Meeting, 272-276. Santa Monica, CA: Human
Factors and Ergonomics Society.
45. Bolstad, C.A. & Endsley, M.R. (in press). Measuring
Shared and Team Situation Awareness in the U.S.
Army’s Future Objective Force. Military Psychology.
46. Jones, D.G. & Endsley, M.R. (2000). Examining the 57. Foltz, P.W., Bolstad, C.A., Cuevas, H.M., Franzke,
Validity of Real-Time Probes as a Metric of Situation M., Rosenstein, M., & Costello, A.M. (2008).
Awareness. Proceedings of the 14th Triennial Measuring Situation Awareness through Automated
Congress of the International Ergonomics Communication Analysis. In M. Letsky, N. Warner,
Association and the 44th Annual Meeting of the S. M. Fiore, & C. Smith (Eds.), Macrocognition in
Human Factors and Ergonomics Society. Santa Teams. Aldershot, England: Ashgate.
Monica, CA: Human Factors and Ergonomics 58. Xie, B. (2007). Information Technology Education for
Society. Older Adults as a Continuing Peer-Learning
47. Bolstad, C.A., Cuevas, H.M., Costello, A.M., & Process: A Chinese Case Study. Educational
Rousey, J. (2005). Improving Situation Awareness Gerontology, 33(5), 429-450.
through Cross-Training. Proceedings of the 49th 59. Endsley, M. R. (1993). Situation Awareness and
Annual Meeting of the Human Factors and Workload: Flip Sides of the Same Coin. Seventh
Ergonomics Society, 2159-2163. Santa Monica, CA: International Symposium on Aviation Psychology.
Human Factors and Ergonomics Society. Columbus, OH: Department of Aviation, The Ohio
48. Jones, D.G. & Endsley M.R. (1996). Sources of State University.
Situation Awareness Errors in Aviation. Aviation, 60. Fracker, M. L., & Davis, S. A. (1990). Measuring
Space and Environmental Medicine 67(6), 507-512. Operator Situation Awareness and Mental Workload.
49. Rodgers, M.D., Mogford, R.H. & Strauch, B. (2000). Paper presented at the Fifth Mid-Central
Post-Hoc Assessment of Situation Awareness in Air Ergonomics/Human Factors Conference, Dayton,
Traffic Control Incidents and Major Aircraft OH.
Accidents. In M.R. Endsley & D.J. Garland (Eds.), 61. Venturino, M., Hamilton, W. L., & Dvorchak, S. R.
Situation Awareness Analysis and Measurement, (1989). Performance-Based Measures of Merit for
73-112. Mahwah, NJ, Lawrence Erlbaum. Tactical Situation Awareness. Situation Awareness
50. Jones, D.G. (2000). Subjective Measures of in AERO 4/5. Copenhagen, Denmark: NATO –
Situation Awareness. In M.R. Endsley & D.J. AGARD.
Garland (Eds.), Situation awareness analysis and
measurement, 113-128. Mahwah, NJ, Lawrence
Erlbaum.
51. Matthews, M. D., Beal, S. A., & Pleban, R. J. (2002).
Situation Awareness in a Virtual Environment:
Description of a Subjective Measure. (Research
Report 1786). Alexandria, VA: U.S. Army Research
Institute for the Behavioral and Social Sciences.
52. Endsley, M.R. (1990). Predictive Utility of an
Objective Measure of Situation Awareness.
Proceedings of the Human Factors Society 34th
Annual Meeting, 41-45. Santa Monica, CA: Human
Factors and Ergonomics Society.
53. Strater, L.D., Endsley, M.R., Pleban, R.J., &
Matthews, M.D. (2001). Measures of Platoon Leader
Situation Awareness in Virtual Decision Making
Exercises (No. Research Report 1770). Alexandria,
VA: Army Research Institute.
54. Endsley, M.R. & Jones, W.M. (1997). Situation
Awareness, Information Dominance, and Information
Warfare (No. AL/CF-TR-1997-0156). Wright-
Patterson AFB, OH: United States Air Force
Armstrong Laboratory.
55. Bolstad, C.A., Cuevas H.M., Gonzalez, C., &
Schneider, M. (2005). Modeling Shared Situation
Awareness. Proceedings of the 14th Conference on
Behavior Representation in Modeling and Simulation
(BRIMS), Los Angeles, CA.
56. Bolstad, C.A., Foltz, P., Franzke, M., Cuevas, H.M.,
Rosenstein, M., & Costello, A.M. (2007). Predicting
Situation Awareness from Team Communications.
Proceedings of the 51st Annual Meeting of the
Human Factors and Ergonomics Society. Santa
Monica, CA: Human Factors and Ergonomics
Society.
CONTACT

Cheryl A. Bolstad is a Senior Research Associate with

SA Technologies. She has a bachelor's degree in
Computer Applications in Psychology and a Master's
degree in Cognitive Psychology. She completed her
doctoral degree in Psychology at North Carolina State
University, specializing in cognition and aging. Dr.
Bolstad has over 15 years of experience as a human
factors engineer. Her recent research has focused on
developing methods for supporting team situation
awareness in distributed systems and developing
training systems for supporting situation awareness.
Address correspondence to: Cheryl A. Bolstad, Ph.D.,
SA Technologies, 76 Lillian Court, Forest Hill, MD
21050, email: cheryl@satechnologies.com.

DEFINITIONS

Situation awareness: Perception of the elements in the

environment within a volume of time and space, the
comprehension of their meaning and the projection of
their status in the near future [2].

ACRONYMS AND ABBREVIATIONS

ACASA: Automated Communication and Situation

Awareness

ACC: adaptive cruise control

ASR: automatic speech recognition

GDTA: goal directed task analysis

GPS: Global Positioning System

PSAQ: Post-Trial Participant Subjective Situation

Awareness Questionnaire

SA: situation awareness

SAGAT: Situation Awareness Global Assessment

Technique

View publication stats