PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 53rd ANNUAL MEETING—2009 1

EFFECTS OF AUTOMATION TYPES ON AIR TRAFFIC CONTROLLER

SITUATION AWARENESS AND PERFORMANCE

Arathi Sethumadhavan
Texas Tech University

The Joint Planning and Development Office has proposed the introduction of automated systems
to help air traffic controllers handle the increasing volume of air traffic in the next two decades
(JPDO, 2007). Because fully automated systems leave operators out of the decision-making loop
(e.g., Billings, 1991), it is important to determine the right level and type of automation that will
keep air traffic controllers in the loop. This study examined the differences in the situation
awareness (SA) and collision detection performance of individuals when they worked with
information acquisition, information analysis, decision and action selection and action
implementation automation to control air traffic (Parasuraman, Sheridan, & Wickens, 2000).
When the automation was unreliable, the time taken to detect an upcoming collision was
significantly longer for all the automation types compared with the information acquisition
automation. This poor performance following automation failure was mediated by SA, with lower
SA yielding poor performance. Thus, the costs associated with automation failure are greater when
automation is applied to higher order stages of information processing. Results have practical
implications for automation design and development of SA training programs.

To meet the increasing demands of air traffic in the
next two decades, the Joint Planning and Development
Office has proposed initiatives to modernize the U.S.
National Airspace System (JPDO, 2007). That proposal
involves introduction of automated systems that would help
air traffic controllers to cope with increasing traffic
demands. Though fully automated systems offer several
Copyright 2009 by Human Factors and Ergonomics Society, Inc. All rights reserved. 10.1518/107118109X12524440832700
when adaptive automation was applied to information
acquisition compared to information analysis, decision
making, and action implementation. Low LOAs have also
been shown to promote faster recovery following
automation failure (e.g., Kaber, Onal, & Endsley, 2000;
Sarter and Schroeder, 2001). For example, using a military
decision-making task, Crocoll and Coury (1990) showed

benefits, a potential consequence of such systems is the
out-of-the-loop performance (OOP) problem, which refers
to the reduced ability of operators working with fully
automated systems to perform tasks manually following an
automation failure (e.g., Billings, 1991; Wiener & Curry,
1980). Some researchers attribute loss of operator situation
awareness (SA) to be the main factor responsible for the
OOP problem, bred by overreliance on, and passive
monitoring of automation (e.g., Endsley & Kiris, 1995).
Therefore, it is important that automated systems be
designed that promotes SA and keeps air traffic controllers
in control and in the loop.
One approach to improve SA and reduce the OOP
problem is to use varying levels of automation (LOA). The
central concept of LOA is that automation is not an all or
none phenomenon, but instead can be implemented at
various levels (e.g., Billings, 1991; Wiener & Curry, 1980).
More recently, Parasuraman, Sheridan, and Wickens (2000)
proposed the PSW model of automation. According to this
model, automation can be applied to different stages of
information processing: information acquisition,
information analysis, decision and action selection, and
action implementation. In addition, different LOAs can be
applied to each of these information processing stages.
Low LOAs have been shown to promote operator SA.
For example, using a simulated air traffic control
environment, Kaber, Perry, Segall, McClernon, and Prinzel
(2006) showed that individuals were more accurate in
answering perception-based questions about aircraft states
Downloaded from pro.sagepub.com at PENNSYLVANIA STATE UNIV on May 12, 2016
that when the automation failed, participants who received
only status information (low LOA) performed better than
those receiving recommendations (high LOA).
EXPERIMENT
Purpose
The primary purpose of this study was to examine the
differences in the SA and collision detection performance
of individuals when they controlled air traffic using
different types of a collision detection system. The PSW
model of automation (Parasuraman et al., 2000) was
employed in this study. In the information acquisition
condition, an automated aid supported basic sensory
processes by cueing the altitude of aircraft using color. In
the information analysis condition, the aircraft pairs that
were projected to be in conflict were highlighted in the
airspace. In the decision and action selection condition, a
recommendation to resolve an upcoming conflict between
aircraft pairs was provided. Finally, in the action
implementation condition, the automation detected as well
as resolved upcoming conflicts and provided feedback to
the participants. Though prior research has examined the
extent to which human performance varies under different
automation conditions in domains such as military (e.g.,
Crocoll & Curry, 1990), robotics (Kaber et al., 2000), and
driving (e.g., Endsley & Kiris, 1995), there has been
limited work (e.g., Kaber et al., 2006) conducted to
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 53rd ANNUAL MEETING—2009
examine the effect of types and levels of automation in the
air traffic control context. Examining air traffic controller
performance consequences under different automation
conditions is important to design automated aids for the
future airspace system.
This study also examined whether the relationship
between the type of automation and performance following
automation failure was mediated by SA. Though prior
studies (e.g., Endsley & Kiris, 1995, Kaber et al., 2000)
have attributed the loss of SA to be the central factor
responsible for the OOP problem, these studies have failed
to demonstrate whether the loss of SA is responsible for
performance degradation following automation failure.
Method
Participants. Seventy- two individuals (27 men and 45
women) ranging in age from 18 to 30 years (M = 21.11, SD
= 3.62) participated.
Simulation Software. Air traffic scenarios were
simulated using a modification of the Air Traffic Scenarios
Test (ATST), a low fidelity simulation of the radar display
(See Figure 1). The airspace consisted of four sector gates to
neighboring sectors (A, B, C, and D) and two airports (x, y).
E, F, G, H, and I were waypoints, through which an aircraft
had to pass to get to its final destination. An aircraft could fly
at three possible speeds (Fast, Medium, or Slow) and at any
of the three altitudes (1, 2, or 3). Aircraft were identified by
call signs (e.g., 5, 13).
Figure 1. Snapshot of the ATST airspace.
The Air Traffic Control Task. Participants were
instructed to avoid collisions during the scenarios.
Collisions occurred when two aircraft flying at the same
altitude were within 5 miles of each other. They were
instructed to avoid collisions by making an altitude
separation. Participants were told to press a button labeled
‘conflict’ on the ATC display when they detected a
potential collision. They were also asked to say aloud the
Downloaded from pro.sagepub.com at PENNSYLVANIA STATE UNIV on May 12, 2016
2
call signs of the aircraft involved in the upcoming collision.
The advance notification time, which referred to the time a
potential collision was scripted to occur in the scenario
minus the time the participant detected the potential
collision by pressing the conflict button (Metzger &
Parasuraman, 2005) was recorded by the ATST simulator.
The participants were also instructed to follow the
rules of the simulation environment. They were told to land
aircraft at airports at speed slow and altitude 1 and exit
aircraft through sector gates at speed fast and level 3. In
addition, participants were also instructed to activate the
aircraft that appeared in their airspace as soon as possible in
order to reduce handoff delays.
Secondary Task. In addition to the primary task of
controlling air traffic, participants were given an additional
task of monitoring a weather display. They were told that
the weather monitoring task was secondary to the primary
task of controlling traffic. Consistent with prior research
(e.g. , Kaber & Endsley, 2004; Metzger & Parasuraman,
2005), the purpose of introducing this task was to induce
overreliance on the automated aids provided in the air
traffic control task as well as to be representative of a
multi-task environment like air traffic control where
controllers have to exhibit good performance in multiple,
concurrent tasks. This task was presented on a laptop
placed adjacent to the radar display. A series of numbers
between 100 and 199 appeared at 3-second intervals.
Participants were told that a number that was less than 130
or greater than 170 represented a wind shear warning.
Hence, they were told to press the spacebar key on the
laptop when they saw a number less than 130 or greater
than 170. Not more than 3 adjacent numbers were signals
(requiring a key press) or noise. The hit-to-signal ratio (the
number of hits/ (number of hits + misses) and the number
of false alarms were recorded.
Experimental Design. The independent variable in this
study was the automation type, which served as the
between subject grouping variable: information acquisition,
information analysis, decision and action selection, and
action implementation.
In the information acquisition condition, an automated
aid displayed aircraft that were flying at the same altitude
in the same color in the ATC display. Specifically, all the
aircraft flying at altitude 3, 2, and 1 were presented in the
airspace in green, pink, and blue respectively. Research in
visual search tasks have shown that color can be
preattentively processed (e.g., Treisman, 1986), thereby
helping controllers to assess altitude, an important piece of
information that needs to be monitored to project whether
collisions would occur, without having to pay focused
attention to datablocks (e.g., Johnston, Horlitz, &
Edminston, 1993). Thus, a moderate LOA (e.g.,
Parasuraman et al., 2000) that highlights important
information (i.e., altitude) was applied to the information
acquisition stage of information processing. In the
information analysis condition, an automated aid
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 53rd ANNUAL MEETING—2009
highlighted the aircraft pairs that were projected to be in
conflict. Specifically, a high LOA (e.g., Parasuraman et al.,
2000) that integrates information from multiples sources
(i.e., altitude, heading, location, and destination) and
projects future events (i.e., collisions) was applied to the
information analysis stage of information processing. In the
decision and action selection condition, an automated aid
recommended the course of action (i.e., altitude separation)
to resolve a potential collision. Specifically, level 4 in the
Sheridan and Verplank (1978) LOA taxonomy (i.e.,
‘Computer suggests one alternative’) was applied to the
decision and action selection stage of information
processing. In the action implementation condition, an
automated aid detected potential collisions and
autonomously resolved these collisions by making an
altitude separation. Specifically, level 7 in the Sheridan and
Verplank (1978) LOA taxonomy (i.e., ‘Computer executes
selected option and informs the human’) was applied to the
action implementation stage of information processing.
The dependent variables were SA, ATC task
performance, and secondary task performance. SA was
measured using a modification of the Situation Awareness
Global Assessment technique (SAGAT; Endsley, 1995).
The simulation froze twice during the scenario and the
participants were asked to recall the location, call sign,
altitude, heading, and destination of all the aircraft present
in the airspace. Aircraft speed was omitted from the
analysis because the aircraft flew at speed fast at virtually
all times. The responses during the first and second
SAGAT freezes were averaged to obtain an overall
proportion accuracy score for location, call sign, altitude,
heading, and destination. The proportion accuracy scores
for all the aircraft attributes were averaged to obtain the
total SA score.
The air traffic control performance variable of interest
was the advance notification time (Metzger &
Parasuraman, 2005). Advance notification time was
computed as the time a planned collision would occur (or
would have occurred) in the scenario minus the time the
participant reported the collision by pressing the conflict
button. In addition to performance in the air traffic control
task, the performance in the secondary task was also
assessed. Specifically, the hit-to-signal ratio and the
number of false alarms were recorded.
Procedure. The experiment lasted for approximately 3
hours. Participants first watched an instructional video that
demonstrated the layout of the airspace and then
memorized the map of the airspace. Thereafter, they
watched instructional videos that demonstrated the rules to
control traffic in the ATST simulator. They then completed
manual training in the air traffic control task. During this
phase, they first learned to control air traffic and
subsequently to perform the air traffic control task in
conjunction with the secondary task. The purpose of
providing manual training to the participants before
assigning them to work with an automated aid was to help
Downloaded from pro.sagepub.com at PENNSYLVANIA STATE UNIV on May 12, 2016
3
them develop appropriate mental models that would help
them to take over manual control when an automation
failure occurs (e.g., Kessel & Wickens, 1982; Moray,
1986).
After completing the manual test scenarios,
participants were randomly assigned to one of the
automation types. The participants first received
instructions on the functionalities of the automated aid.
They were informed that the automated aids were highly
reliable, though not 100% reliable and that even if the
automation fails it was their responsibility to monitor the
aircraft in their airspace and avoid collisions. They then
completed several training scenarios in which they were
assisted by the automation. During these scenarios, the
automation did not fail so as to build up participants’ trust
on the automated aids. This was consistent with the
methodology adopted in earlier studies (e.g., Kaber et al.,
2000; Rovira, McGarry, & Parasuraman, 2007).
After completing the training scenarios, participants
completed a 15-minute experimental scenario. This
scenario involved a total of 12 scripted collisions. The
different automation types were perfectly reliable in
detecting the first 11 upcoming collisions. The automated
aids failed to detect the 12th scripted collision. That is, in
the information acquisition condition, the aid failed to
present the data block of one of the aircraft that would be
involved in an upcoming collision in the correct color. In
the information analysis condition, the aid failed to
highlight the aircraft pairs that would be involved in the
upcoming collision. In the decision and action selection
condition, the aid failed to provide the recommendation to
resolve the collision. Finally, in the action implementation
condition, the aid failed to detect and resolve the collision.
The time taken by participants working with different
automation types to detect the 12th scripted collision (i.e.,
advance notification time) following the failure of the
automation was recorded. The scenario ended 8 seconds
after the occurrence of the 12th scripted collision.
SA of the participants was assessed twice during the
scenario. Both SA assessments were conducted prior to the
occurrence of the automation failure so as to determine the
extent to which pre-automation failure SA mediated the
relationship between automation type and collision
detection performance following automation failure.
Results and Discussion
Air Traffic Control Performance Following
Automation Failure. A one-way Analysis of Variance
(ANOVA) with the type of automation as the between
subject variable was conducted on advance notification
following automation failure. The effect of automation type
on advance notification time was significant, F (3, 68) =
16.507, p < .001, ηp2= 0.421. Tukey’s HSD analysis
showed that individuals working with information
acquisition automation were significantly faster in detecting
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 53rd ANNUAL MEETING—2009 4

the upcoming collision compared to individuals working analysis automation, decision and action selection
with all the other automation types, following the automation, and action implementation automation were
automation failure (See Figure 2). No significant 0.514 (SD =.101), 0.397 (SD =.085), 0.416 (SD =.085), and
differences in the advance notification time were observed 0.383 (SD =.125), respectively.
between the individuals working with the information In order to determine whether the relationship between
analysis automation (95% CI = 18.842, 37.368), decision automation type and the advance notification time
automation (95% CI = 7.173, 25.699) and action following automation failure was mediated by SA, the
implementation automation (95% CI = 4.907, 23.433). methods developed by Baron and Kenny (1986) were used.
However, a less conservative comparison between the Four linear regression analyses were conducted to test for
groups using planned contrasts indicated that individuals in mediation: automation type predicting SA, SA predicting
the information analysis condition were significantly faster advance notification time, automation type predicting
in detecting the collision following the automation failure advance notification time, and finally automation type and
than those in the decision and action implementation SA predicting advance notification time (See Figure 3).
conditions. This suggested that automation of information Sobel test (1982) statistics revealed significant p value for
analysis had some benefits over automation of decision the indirect effect of SA as a mediator between automation
making and action execution. type and advance notification time following automation
Effect of Automation Type on Advance Notification Time
failure (z = -2.287, p < .05). Thus, SA mediated the
(Following automation failure) relationship between automation type and advance
notification time following automation failure, with higher
70
SA before automation failure contributing to earlier
Advance Notification Time (in seconds)

60 detection of the collision following automation failure.

50 SA (Mediator)
40

30 β = -.379 ** β = .488***

β = -.6***
20

10 Without
0
Automation mediation Advance
uisi
tion naly
sis
lect
ion
ntat
ion Type β = -.484*** notification time
A cq atio
nA n Se eme
at ion r m A ctio I mpl
Info
rm Inf o
isio
n an
d
Act
io n
With
D ec
mediation
Automation Type
Figure 3. Mediation model of the association between
automation type and advance notification time following
Figure 2. Effect of automation type on advance notification automation failure as mediated by SA. Standardized
time following automation failure. The ideal advance regression coefficients are presented in the figure. ***p <
notification time is 70 s. Error bars indicate + 1 standard .001, **p < .01.
error of the mean.
Secondary Task Performance. A one-way ANOVA on
In short, when the automation failed, individuals the hit-to-signal ratio was significant, F (3, 68) = 12.087, p <
working with information acquisition automation detected .001, ηp2 = 0.348. Tukey's HSD analysis showed that
the upcoming collision earlier in comparison to the other individuals working with information analysis (M = 0.906,
automation aids. Thus, automation of sensory processing SD = 0.070), decision (M = 0.933, SD = 0.047), and action
had benefits over automation of higher order cognitive implementation automation (M = 0.939, SD = 0.032) had
functions such as prediction generation and decision higher hit-to-signal ratio compared to those working with the
making. This meant that decreasing the level of operator information acquisition automation (M = 0.818, SD = .103).
involvement can prove hazardous if the automation A one-way ANOVA on the number of false alarms
reliability is not 100%.
failed to reach significance, F (3, 68) = 1.167, p > .05, ηp2
SA. A one way ANOVA with type of automation
=0.049. Results of binomial probability calculations showed
as the between subjects variable on the total SA score was
that the number of false alarms for each automation type was
significant, F (3, 68) = 6.304, p < .01, ηp2= 0.218. Tukey’s significantly less than chance probability. These analyses
HSD analysis revealed that individuals working with helped to establish that participants were attending to the
information acquisition automation had higher overall SA secondary task and not merely pressing the spacebar key.
compared to those working with all the other automation The superior performance in the secondary task by
types. The mean proportion accuracy score for total SA for individuals working with information analysis, decision,
the information acquisition automation, information
Downloaded from pro.sagepub.com at PENNSYLVANIA STATE UNIV on May 12, 2016
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 53rd ANNUAL MEETING—2009
and action implementation automation can be attributed to
the overreliance they exhibited on the collision detection
aids provided to them in the air traffic control task, due to
which they had more cognitive resources or chose to
allocate more cognitive resources to the secondary task.
CONCLUSIONS
This study provides recommendations for automation
design for the future National Airspace System by applying
the PSW model of automation. Based on this model, in
order to determine the appropriate degree of automation
that will keep the operator involved, the consequences for
human performance when interacting with automated
systems applied to different stages of information
processing have to be examined (Parasuraman et al., 2000).
This study revealed that when 100% reliable automation
was not provided, automating sensory processing was
beneficial in helping individuals detect an upcoming
collision earlier following automation failure in comparison
to automation of information analysis, decision making,
and action implementation, by improving their SA. In
summary, applying a moderate LOA (i.e., cueing aid) to the
information acquisition stage can be pursued in the future
airspace system as long as the automation reliability is
high. Although applying high LOAs to information
analysis, decision and action selection, and action
implementation stages can provide benefits such as
enhancing performance in secondary tasks that controllers
have to perform, the risk associated with their failure is
high.
LIMITATIONS AND FUTURE DIRECTIONS
Caution should be exercised in applying the results of
this study to the ATC domain. The background of the
participants and the fidelity of the simulation environment
limit my ability to generalize the findings of this study to
the air traffic control domain. Future work should thus be
directed towards examining the costs associated with
automation failure with professional controllers who have
had extensive manual experience in the air traffic control
domain. Future work should also examine whether
explaining the situations under which the automation
algorithm can fail and making automation failures more
salient by providing the intermediate results of the
automation algorithm and designing more intuitive displays
(e.g., Lee & See, 2004; Metzger & Parasuraman, 2005) can
help to reduce the misuse of information analysis, decision,
and action implementation automation.
ACKNOWLEDGMENT
This work was part of the author’s dissertation. The
author would like to thank her academic advisor, Dr. Frank
Durso and the other members in her dissertation committee for
Downloaded from pro.sagepub.com at PENNSYLVANIA STATE UNIV on May 12, 2016
5
their valuable comments. Special thanks to Dr. Chris Wickens
for his comments on this paper.
REFERENCES
Baron, R.M., & Kenny, D. A. (1986). The moderator-mediator variable
distinction in social psychological research: Conceptual, strategic,
and statistical considerations. Journal of Personality and Social
Psychology, 51, 1173-1182.
Billings, C. E. (1991). Human-centered aircraft automation: A concept and
guidelines (NASA Tech. Memorandum 103885). Moffet Field,
CA: NASA Ames Research Center.
Crocoll, W. M., & Coury, B. G. (1990). Status or recommendation: Selecting
the type of information for decision aiding. In Proceedings of the
Human Factors and Ergonomics Society 34th Annual Meeting (pp.
1524-1528). Orlando, FL: Human Factors and Ergonomics
Society.
Endsley, M. R. (1995). Measurement of situation awareness in dynamic
systems, Human Factors, 37, 65-84.
Endsley, M. R. & Kiris, E. O. (1995). The out-of-the-loop performance
problem and level of control in automation. Human Factors, 37,
381-394.
Johnston, J. C., Horlitz, K. L., & Edmiston, R. W. (1993). Improving situation
awareness displays for air traffic controllers. In R. S. Jensen & D.
Neumesiter (Eds.), Proceedings of the 7th International Symposium
on Aviation Psychology (pp. 328-334). Columbus, OH: Ohio State
University.
JPDO (2007). Concept of Operation for the Next Generation Air
Transportation System. Retrieved October 29, 2008, from,
http://www.jpdo.gov/library/nextgen_v2.0.pdf
Kaber, D., & Endsley, M. (2004). The effects of level of automation and
adaptive automation on human performance, situation awareness
and workload in a dynamic control task. Theoretical issues in
Ergonomic Science, 5, 113–153.
Kaber, D. B., Onal, E., & Endsley, M. R. (2000). Design of automation for
telerobots and the effect on performance, operator situation
awareness, and subjective workload. Human Factors and
Ergonomics in Manufacturing, 10, 409-430.
Kaber, D.B., Perry, C. M., Segall, N., McClernon, C. K., & Prinzel, L.J.
(2006). Situation awareness implications of adaptive automation
for information processing in an air traffic control-related task.
International Journal of Industrial Ergonomics, 36, 447-462.
Kessel, C. J., & Wickens, C. D. (1982). The transfer of failure-detection skills
between monitoring and controlling dynamic systems. Human
Factors, 24, 49-60.
Lee, J., & See, J. (2004). Trust in automation: Designing for appropriate
reliance. Human Factors, 46, 50-80.
Metzger, U., & Parasuraman, R. (2005). Automation in future air traffic
management: Effects of decision aid reliability on controller
performance and mental workload. Human Factors, 47, 35-49.
Moray, N. (1986). Monitoring behavior and supervisory control. In K. Boff
(Ed.), Handbook of perception and human performance (pp. 40/1-
40/51). New York: Wiley.
Parasuraman, R., Sheridan, T. B., & Wickens, C. (2000). A model for types
and levels of human interaction with automation. IEEE
Transactions on Systems, Man, and Cybernetics, 30, 286-297.
Rovira, E., McGarry, K., & Parasuraman, R. (2007). Effects of imperfect
automation on decision making in a simulated command and
control task. Human Factors, 49, 76-87.
Sarter, N. B., & Schroeder, B. (2001). Supporting decision making and action
selection under time pressure and uncertainty: The case of in-flight
icing. Human Factors, 43, 573-583.
Sheridan, T. B., & Verplank, W. L. (1978). Human and computer control of
undersea teleoperators. Cambridge, MA: MIT Man-Machine
Laboratory.
Sobel, M. E. (1982). Asymptotic confidence intervals for indirect effects in
structural equations models. In S. Leinhart (Ed.), Sociological
methodology (pp. 290-312). San Francisco: Jossey-Bass.
Treisman, A. (1986). Features and objects in visual processing. Scientific
American, 255, 114–125.
Wiener, E. L., & Curry, R. E. (1980). Flight deck automation: Promises and
problems. Ergonomics, 23, 995-1011.