Applied Ergonomics 100 (2022) 103670

Contents lists available at ScienceDirect

Applied Ergonomics
journal homepage: www.elsevier.com/locate/apergo

Impact of alarm management and automation on abnormal operations: A

human-in-the-loop simulation study
Richard J. Simonson a, *, Joseph R. Keebler a, Elizabeth L. Blickensderfer a, Ron Besuijen b
a
Department of Human Factors and Behavioral Neurobiology, Embry-Riddle Aeronautical University, Daytona Beach, FL, USA
b
Center for Operator Performance, Dayton, OH, USA

A R T I C L E I N F O A B S T R A C T

Keywords: Process control consists of complex states of performance that require a significant amount of operator attention
Process control and skill to manage. Automation and smart alarms can enhance these processes. This study investigated the
Workload effects of alarm management and automation on process control operator workload (NASA-TLX) and perfor­
Flare
mance via material lost through flaring outcomes. Eleven console qualified operators participated in a human-in-
Alarm rationalization
Automation
the-loop, high-fidelity simulation-based training exercise. Three levels of alarm management (no rationalization,
rationalization, and state-based smart alarm) and two levels of automation (no automation and automation)
conditions were implemented using six scenarios. A repeated measures two-way MANOVA indicated that both
alarm management schema and automation significantly affected operator workload and performance. These
results indicate that state-based smart alarm management and automation schemas may assist operators in
reducing workload and material lost through flare release during abnormal operating conditions.

1. Introduction
Manufacturing and other complex product refinement and develop­
ment domains (e.g., mining, oil refining, chemical manufacturing,
pharmaceutical manufacturing) actively utilize process control to
monitor processes underway and react to any abnormal or out-of-control
processes for safety and efficiency purposes (NIST/SEMATECH
e-Handbook of Statistical Methods, 2013). The complexity of these
systems in the recent century has far surpassed the operator’s cognitive
abilities to perceive and attend to all pertinent pieces of information
presented to them (Nochur et al., 2001). During states of abnormality or
out-of-control processes, process controls have been shown to activate
and elicit up to 275 alarms during the event (Hollender and Beuthel,
2007), and in one case over 300 alarms in just 5 min (Strobhar, 2012).
This excess of information presents a multitude of challenges, mainly
due to the limited cognitive abilities of attention (Miller, 1956; Dadashi
et al., 2017). The discrepancy between the amount of information pre­
sented and the amount of information to which individuals can effec­
tively attend leads to increased workloads, human error, and decreases
in efficiency (Stanton et al., 2009). Alarm schemas (e.g., rationalized
and “smart alarming”) (Hollifield, 2010) and automation technologies
have been used to mitigate these challenges in the process control
industry. This paper describes a study that examined the impact of alarm
schema and levels of automation on operator performance and workload
in a high-fidelity simulation of a chemical manufacturing plant.
1.1. Alarm strategy
In terms of alarm strategy, rationalized alarm strategies are those
that review and validate the state of the processes and determine if an
alarm meets the criteria of identifying or warning of an abnormal or out-
of-control process (Beebe et al., 2013; Rockwell Automation, 2017a).
Alarm practices such as rationalization have been shown to be a sig­
nificant source of increased efficiency and time reductions in finding the
true source of alarm activations. Specifically, it has been shown that up
to 80% of alarms during an event are the result of nuisance or bad actor
alarms, which are defined as alarms that activate as an indirect result of
the source (Rockwell Automation, 2017b). These nuisance and bad actor
alarms can divert operator attention and/or interfere with the operator’s
acknowledgment of the source alarm within a reasonable and necessary
time frame. While the positive effects of properly designed and imple­
mented alarm management schemas are apparent, the development of a
system requires careful consideration of the system, possible alarms,
past experiences, and the processes being controlled.

* Corresponding author. 1 Aerospace Blvd, Daytona Beach, FL, 32114, USA.

E-mail address: simonsr1@my.erau.edu (R.J. Simonson).

https://doi.org/10.1016/j.apergo.2021.103670
Received 20 May 2021; Received in revised form 18 November 2021; Accepted 13 December 2021
Available online 21 December 2021
0003-6870/© 2021 Elsevier Ltd. All rights reserved.
R.J. Simonson et al.
A multitude of recommended alarm management policies and
development frameworks exist to assist control process personnel
develop such a schema (e.g., Rockwell Automation, 2017a; Brown,
2003; Veland et al., 2001), the focus of the current project is on smart
alarming. Smart alarming is a type of alarm management that relies on a
state-based philosophy, wherein only alarms that would be applicable to
the state of the process would be activated (Siemens, 2018). Typically,
the research on alarm management focuses on the reduction of the
number of alarms rather than on its impact on human operator perfor­
mance. It is assumed that fewer alarms lead to better performance. One
likely reason for the lack of performance-based research is that to best
assess the impact of alarm design on operators’ performance, a
high-fidelity simulation of the particular manufacturing process is
required. Hence, one aim of the current study was to fill this research
gap through studying smart alarming in a high-fidelity chemical pro­
cessing simulation.
1.2. Automation
Automation techniques in process control have been designed to
reduce the workload that an operator must undertake when responding
to abnormal events. An automated system is defined as “a system that
accomplishes (partially or fully) a function that was previously, or
conceivably could be, carried out (partially or fully) by a human oper­
ator” (Parasuraman et al., 2000, p. 287). Automation operates between a
low and high level of intervention in tasks. Numerous taxonomies that
categorize and classify differences in automation levels exist (e.g.,
Endsley, 1987; Endsley, 1999; Riley, 1989; Sheridan and Verplank,
1978). The taxonomies emphasize that automation should match the
stages of human information processing (Popken and Krems, 2017).
While the benefits of automation are vast and include decreased
workload, increased situational awareness, and improved performance
(Lorenz et al., 2001), the dangers of inappropriate usage/applications of
automation can be devastating (Reddy and Mehta, 2015). A primary
issue with implementing automation into a human-machine system is
that the automation can negatively impact the operators’ understanding
of the system overall, and in turn, have a negative impact on specific task
processes. Negative impacts of automation include requiring excessive
operator monitoring of automated tasks (and, in turn, operator
boredom), operators over-relying on the automated system, operators
having low situational awareness, operators performing out-of-the-loop,
and/or reduced operator trust of the automation (Popken and Krems,
2017). Operators who are performing out-of-the-loop are those who are
unaware of the systems current process state. This operating state is
typically associated with automation, as it provides a means to allow an
operator to pay less attention to various details regarding the process
(Endsley and Kiris, 1995). Research demonstrates that unsuitably high
or low automation usage can cause errors and violations, which in turn,
lead to adverse events (de Visser and Parasuraman, 2011). In fact, these
are the most frequent challenges that occur, either with an improperly
designed performance-feedback loop that generates low operator trust
in automation (i.e., negates the need for it) or overly high operator trust
in automation (i.e., over-reliance on the automated system and infor­
mation). Additionally, another area of research, adaptive automation,
focuses on actively conforming an automated process to the operators’
physical and mental demands (Hancock et al., 2013). While automation,
in general, has been implemented into the process control industry,
adaptive automation has yet to be employed. However, recent research
into adaptive automation within the process control industry provides
evidence that, during complex and information-rich scenarios, operators
performed better and preferred high levels of automation that required
them to progress the scenario by actively participating in the automation
(e.g., forced-choice automation) (Sauer et al., 2011). These results
extend prior beliefs that providing an avenue to invite the operator to
work in-the-loop with automation may yield better results.
On-going research is aimed at how to most effectively implement
2
Applied Ergonomics 100 (2022) 103670
automation within the process control industry (as well as many other
industries), and several recommendations exist for best utilizing auto­
mation. First, current process control tasks are designed around human
cognitive abilities, processes, and their advantages and limitations.
Hence, a key step when implementing automation into the process
control tasks is to evaluate the task structures and change those task
structures as needed to suit both the automation technology, the human
operators, and the interactions between the two (Lee and Seppelt, 2009).
To accomplish this, recommended strategies include the use of Fitts’ list
(Fitts, 1951) to consider human information processing strengths and
weaknesses when assigning tasks to automation/machines versus
human operators, ensure the operator(s) understands the distribution of
responsibilities (i.e., what tasks the automation vs. the operator will
perform), and employ operator training that teaches the operators how
to interact with the automated tasks (Lee and Seppelt, 2006).
The second recommendation is the need to account for the human
operators’ characteristics and match the automation to those charac­
teristics (Lee and Seppelt, 2006). That is, instead of applying automation
to an omnibus task, it is recommended to analyze or break down the task
in terms of interactions with the stages of human information processing
and, in turn, design the automation to best accommodate human
cognitive performance. For example, Lee et al. (1999) provide evidence
that notification-based automation alerts as a way to capture the oper­
ator’s attention were more effective than command-based alerts at
increasing the performance and safety of human drivers. These infor­
mation interpretation messages are shown to be more effective at
providing operators a greater mental model of the situation they are
engaged in as compared to action selection commands (Lee and Seppelt,
2006).
A third recommendation for utilizing automation is to ensure auto­
mated systems provide appropriate feedback to the operators. Adequate
feedback is essential for operators to develop accurate mental models of
the situation and to prevent operators from out-of-the-loop performance
(Lee and Seppelt, 2006). Display strategies that include visual forms of
data relationships, use of visual features that represents said data, and
providing data and its visualized forms in meaningful contexts are
proven methods of providing intuitive feedback to human process con­
trol operators (Guerlain et al., 2002).
Finally, automation should match the mental model of the operator.
Lee and Seppelt (2006) provide the example that cruise control and
assisted breaking technologies match the driver’s mental model of
driving as they perform similar reactions and feedback that the driver
would expect. They exemplify that an automated braking system slows
the car down when an obstacle is sensed in front of the car, indicating to
the driver that they can enter to control-loop and brake for a more im­
mediate and controllable reaction if needed (See Table 1 for a summary
of recommendations).
An example of the implementation of automation within process
control industries that follow these suggestions is the usage of safe-park.
Safe-park consists of a series of processes that drive the system to a
suitable point wherein the out-of-control process can be isolated. In this
investigation, we operationalize safe-park as shutting down and
Table 1
Recommended methods of implementing automation.
Recommendation Example
Use Fitts’ list. Information processing at the information
acquisition, analysis, action selection and
implementation stages.
Match automation and human Use notification-based alerts rather than
characteristics. command based-alerts.
Provide appropriate feedback. Utilize dashboards to provide operators with up-
to-date and accurate information regarding
process status and automation decisions.
Match automation and operator Utilize automated vehicle braking systems that
mental models. invite the operator back into the loop.
R.J. Simonson et al.
isolating the out-of-control process completely. The alarm management
and automation schemas utilized in the investigation were manipulated
to determine if any significant differences were present in operator
performance and workload outcomes.
1.3. Operator performance in ethylene production
Flaring is considered to be a natural, although undesirable, process
within the gas production and processing procedure. Upon
manufacturing startup, shutdown, and during the manufacturing stages,
it is known that some of the product will be released through collection
headers where it is later burned in flare stacks (Ohio, 2014). In most
cases, the resulting flaring is due to production testing and gas man­
agement, however, in some cases, flaring is a part of an emergency,
where some abnormal or adverse event causes or necessitates the release
of the product to the flare.
While flaring is often a natural and controlled process during normal
operations, these emergency scenarios can release flare at possibly un­
controlled rates depending on the severity of the emergency for a
duration of time until the offending issue is addressed. During these
events, the processing plant is exposed to increased hazards and lose
their product. Additionally, potential greenhouse gasses in the form of
carbon dioxide (CO2) and water vapor (H2O) are released into the sur­
rounding ecosystem (Emam, 2015).
1.4. Purpose of the study
Alarm management schemas and automation are widely used within
the process control industries to prevent, manage, and mitigate out-of-
control and abnormal processes. Published literature, industry guides,
and government ISO’s provide guidance on how to effectively imple­
ment automation and alarming solutions, however, each industry and
process may require novel and/or unique modifications for a successful
application. Hence, a portion of process control research has been
dedicated to the investigation of these effects and the implication of the
results for the applicable industries. The purpose of the current study
was to conduct a high-fidelity, simulation-based, human-in-the-loop
study to quantify the effects of alarm design and automation
Fig. 1. Operator monitoring a simulated control room display.
3
Applied Ergonomics 100 (2022) 103670
techniques on operator performance and workload in ethylene
manufacturing. Based on prior literature, the predictions were: 1) the
use of rationalized and smart alarm schemas would reduce operator
perceived workload and the amount of flaring as compared to a non-
rationalized alarm design, and 2) the use of automation (i.e., a safe-
park application) would reduce both operator perceived workload and
the amount of flaring as compared to without the use of the automation.
2. Method
2.1. Apparatus
The quality improvement study focused on the indoor, control panel
operating tasks for the finishing side of ethylene manufacturing. A high-
fidelity, dynamic simulator modeled by Corys (2015) was used (see
Fig. 1). The model was for the finishing side of the process.
Each set of monitors represented in Fig. 1 are set to display specific
information during a simulation-based training scenario in ethylene
production. The Upper two monitors visualize fixed emergency alarms
that are always on display. The first monitor on the left in the middle
displays a video of the flare stack while the second to the left middle
monitor displays the equipment being used in the process. The third
from the left middle monitor displays alarm summary and trends while
the fourth and fifth from the left middle displays are used to monitor
process trends during steady-state operations. All four monitors on the
bottom are used to display an overview of the process and allow for the
operator to display visualizations of the process in question. During the
conditions experienced as a part of this investigation, all monitors were
on and displayed the information as if it were a real scenario. However,
to successfully complete the scenarios participants were only required to
use the bottom left monitor to view a graphic of the process.
2.2. Operators
Eleven male finishing side operators currently employed at an
ethylene refinery in Canada participated. All operators were shift
workers on a five-week rotating schedule and were scheduled for their
required simulation-based training. As all operator participants were
R.J. Simonson et al.
active panel operators with an average experience of 13.18 (SD = 6.38)
years, they were familiar with the equipment, processes, and procedures
specific to the plant and specific to their job. Prior to the start of the
scenario operators indicated their levels of alertness on a Likert-type
scale from one to seven, one being exhausted and seven being more
alert than usual. On average, the operators reported feeling relatively
average energy levels (M = 3.7, SD = 0.64). Additionally, the average
participant’s previous active shift (i.e., previous day of work) was 175
days (SD = 142.2) prior to his training session.
2.3. Simulated scenario
The study focused on the indoor control panel operating tasks for the
finishing side of ethylene manufacturing. The simulation began with a
malfunction of the propylene refrigeration compressor (i.e., “K601
trip”), and hence the plant was in “upset” mode and not steady-state.
When the refrigeration compressor tripped, the control room operator
worked to isolate the plant, and this included closing valves, shutting off
pumps, and flaring (i.e., burning off excess gases). As the operator was
isolating the plant, a valve malfunction occurred. The operators’ task
was to detect the valve malfunction while continuing to resolve the
K601 trip (see Table 2).
2.3.1. Valve malfunctions
To avoid learning effects, six separate scenarios (one scenario per
condition) were used in the study, and each lasted 10–45 min. The
scenarios were experiment condition-specific, which meant that all op­
erators experienced the same scenario for a particular experiment con­
dition. In all scenarios, a valve malfunction occurred during the standard
procedures following the K601 trip. The exact valve that malfunctioned
was different in each scenario. Five scenarios contained valves that
impacted vessel levels and, in one scenario, the valve controlled pres­
sure. When the operator closed the particular valve, a malfunction
occurred such that the valve appeared to be closed on the operator’s
display, but it remained open in the (simulated) plant (Fig. 2.). In one
scenario, the malfunction differed slightly in that a valve that was nor­
mally closed, and appeared closed on the display, drifted to 30% open in
the first minute following the “K601 trip”. A description of the source
alarms for each scenario is available in Table 2.
2.4. Experimental design
The study compared three types of alarm design. The three types
were No-rationalization (which yielded approximately 250 alarms),
Rationalized (which yielded approximately 150 alarms), and Smart/
State-based alarms (which yielded approximately 5 alarms). These
alarming conditions were considered separately to replicate the oper­
ating environment of our simulated environment. In this case, a smart
alarm and rationalized alarm strategy cannot co-exist within the same
scenario. Thus, they were used in comparison to one another. These
alarm levels were designed by a subject matter expert (SME) in the
process control operations domain with over 30 years of experience who
consulted with a Ph.D. level Industrial Engineer with 30 years of expe­
rience working in process control. The SME developed each alarm level,
other than no rationalization, to replicate the processes that would occur
Table 2
Scenario descriptions.
Scenario Conditions Source alarms
1 Rationalization W/Automation Two vessel and one valve alarm
2 Rationalization W/O Automation Two vessel pressure alarms
3 Smart Alarm W/Automation One exchanger alarm
4 No Rationalization W/Automation Two vessel and one valve alarm
5 No Rationalization W/O Automation One exchanger alarm
6 Smart Alarm W/O Automation One vessel alarm
4
Applied Ergonomics 100 (2022) 103670
during a real instance of each failure. For alarm rationalization, the SME
would evaluate any alarm that would appear during the simulation and
removed any alarm that was not justified based on their expert knowl­
edge of the facilities operation and the processes being simulated. For
state-based alarms, the SME removed any alarm that was not related to
the operating state of the facility and then repeated the rationalization
process from the rationalization condition.
The study compared two levels of automation: presence or absence of
the safe-park application. With the safe-park, operators would perform
approximately 8 moves in responding to the simulated events. Without
the safe-park app, operators would perform approximately 120 moves in
response to the simulated events.
Combining the two independent variables resulted in a 3 × 2 fully
within (repeated measures) quasi-experimental study with six study
conditions. Thus, operators experienced three separate, simulated sce­
narios which included the three alarm designs (no-rationalization,
rationalized, and smart-alarming) in conjunction with using the safe-
park application. Additionally, the operators experienced separate,
simulated scenarios with the three alarm designs (no-rationalization,
rationalized, and smart-alarming) when not using the safe-park app.
2.5. Measures
The measures included a demographic data form, the NASA-TLX, and
amount of material released via the flare.
2.5.1. Demographics form
A demographic survey collected background information about
operator age, experience, and level of fatigue.
2.5.2. Workload (NASA-TLX)
The NASA-TLX (Hart, 1986; Hart and Staveland, 1988) is a subjec­
tive questionnaire and psychometrically validated to measure workload
on six different constructs (mental, physical, temporal, performance,
effort, and frustration). When completing the NASA-TLX, the operator
placed an “X” on the continuum for each separate construct. Following
data collection, using the scale hash marks, the researchers converted
the operator’s “X” marks to numerical form. The scale ranged from 1
(low workload) to 20 (high workload). Thus, an X on the low end of the
scale would be a low number, an X in the middle of the scale would be
equal to “10”, and an X on the high end of the scale would be converted
to a higher number. Once the conversions were made for each operator,
the values for each scale were summed together. For each administra­
tion of the NASA-TLX, each operator had one combined NASA-TLX score
that ranges from 6 to 120. The operators completed the TLX after each of
six simulated scenarios, which resulted in six unique TLX scores per
operator (one per scenario).
2.5.3. Flare release
The amount of flaring (i.e., the amount of excess chemicals released)
was recorded by the simulator’s historian function. In the actual plant,
after K601 trips, an application reduces furnace feed by 22% within 3
min. For consistency in the current study, the feed rate was left at that
level, although in a non-simulated K601 trip event, the feed rate would
have been reduced further by operations. The total flare released was
collected by importing historian data into a Microsoft Excel file in 1-min
averages. The averages were then totaled in flare megagram per hour
over the period of time between the first alarm onset and when the
operator identified the anomaly.
2.6. Procedure
Prior to the operators’ scheduled training days, the simulation
specialist explained that an alarm design evaluation was underway. The
operators were informed that, while the simulation exercises in the
evaluation were part of their regular operator training, their data would
R.J. Simonson et al.
Fig. 2. Valve malfunction process.
be depersonalized and investigated at the aggregate level and not used
as job-performance indicators for personnel assessment. Additionally, a
partial counter-balance technique was employed such that a Latin-
square procedure was used to develop a counterbalanced order of con­
ditions/scenarios where operators were randomly assigned to the
particular order of scenarios (Goodwin and Goodwin, 2012).
Upon arriving for the training day, each operator first completed the
demographic questionnaire. Next, the operator performed the first of the
six scenarios. While the operator performed the simulated scenario, the
simulation specialist recorded the time the participant used to complete
the task and the time to detect the anomaly.
After the completion of each scenario the operator completed the
NASA-TLX questionnaire. The simulation specialist then debriefed the
operator about the scenario, and the operator was given a short break of
about 30 min. This process repeated until the operator had completed all
six scenarios. At this time, the simulation specialist provided a full
debrief of the evaluation. The total time was approximately 8 h. As the
training was part of required training, operators were compensated with
their typical training day compensation. After each operator completed
the scenarios, the simulation specialist collected the automatically
recorded data from the simulator and transferred it to a data recording
device.
3. Results
The multivariate omnibus analysis indicated that there were large
main effects on the dependent variables for both alarm (F (4,7) = 7.11, p
= .013, η2p = .80) and automation (F (2,9) = 13.90, p = .002, η2p = .76).
No interaction effects were found (F (4,7) = 2.83, p = .109, η2p = .62)
(see Table 3). Although the effect size of the interaction effect was high
(η2p = .62), the interaction effects power of .474 was insufficient to
detect any effect (Cohen, 1988). To better understand the intricacies of
the main effects, post-hoc analyses were used to determine the indi­
vidual effects of automation and alarming condition on workload and
released flare amount. Correlational values between workload and
performance were assessed to understand the relationship between the
two variables within their individual scenarios. Our results indicate that
scenario 1, scenario 2, and scenario 4 yielded the largest correlational
values (r = 0.458, 0.430, 0.656) respectively while scenario 3, scenario
5, and scenario 6 yielded the lowest correlational values (r = − 0.126,
0.128, − 0.229) respectively. See Fig. 3 for a graphical representation of
these results.
3.1. Flare
A post-hoc analysis of variance on the dependent variable flare
Table 3
Omnibus two-way multivariate analysis of variance (MANOVA).
DV F Hypothesis (Error) df p η2p power
Alarm 7.11 4(7) .013 .80 .88
Automation 13.90 2(9) .002 .76 .98
Alarm * Automation 2.83 4(7) .109 .62 .47
5
Applied Ergonomics 100 (2022) 103670
released was conducted to determine how the alarm and automation
conditions affected it. The analysis indicated that both the alarm (F(2,
20) = 8.66, p = .002, η2p = 0.46) and automation (F(1, 10) = 26.06, p <
.001, η2p = 0.72) resulted in significantly different values of flaring such
that the smart alarm and non-rationalized alarm conditions released the
least amount via the flare released, while the rationalized alarm released
the most. There was no significant difference in the amount of flare
released between the smart alarm and non-rationalized alarm condi­
tions. Additionally, the condition with the safe park automation resulted
in a significantly lower flare release than the condition without auto­
mation (Table 4.). Further evaluation via pairwise comparisons of the
mean differences by way of t-tests of these effects and their results are
presented in Table 5.
3.2. Workload
A post-hoc analysis of variance was also conducted on the dependent
variable workload to determine how the alarm and automation condi­
tions affected it. The analysis indicated that both the alarm (F(2, 20) =
17.43, p < .001, η2p = .64) and automation (F(1, 10) = 21.72, p = .001,
η2p = .69) conditions had significant differences in average reported
workload. These results indicated that the smart alarm resulted in
significantly less reported workload than the rationalized and non-
rationalized alarms whereas no significant difference was found be­
tween rationalized and non-rationalized alarms (Table 6.). Additionally,
the condition with the safe park automation resulted in a significantly
lower workload than the condition without automation. Further evalu­
ation via pairwise analyses of mean differences by way of t-tests is
available in Table 7.
4. Discussion
In summary, this study investigated the effects of alarm design and
automation on operator performance and workload in a high-fidelity,
ethylene manufacturing simulation. The interventions in this experi­
ment both significantly impacted the outcomes of the amount of mate­
rial released by flare and operator perceived workload. Specifically,
operators indicated lower perceived workload and the system released
less material to the flare when using the safe park automation compared
to conditions without it. Additionally, operators reported lower
perceived workload when using smart alarming versus the rationalized
or non-rationalized alarm schemas. These statistically significant re­
ductions in the performance variables measured in this experiment
presents further evidence that implementing alarm management sche­
mas, specifically smart alarming, and automation techniques to assist
human operators improves system performance.
Consistent with prior evidence the implementation of automation
technologies reduced workload and flare release levels regardless of the
type of alarm system used (e.g., non-rationalized, rationalized, and
smart). These results are in clear alignment with the body of literature
indicating that well-designed automation technology can lower human
workload and improves performance (e.g., Endsley, 1999; Kaber and
Endsley, 2004). We posit that this is due to the safe-park’s direct effect
R.J. Simonson et al. Applied Ergonomics 100 (2022) 103670

Fig. 3. Standardized workload and flare by alarm and automation condition.

Table 4 Table 6
Flare released (Mg) Descriptive statistics by condition. Perceived workload (NASA-TLX; 0–120) Descriptive statistics by condition.
Safe-Park Automation No Automation Total Safe-Park Automation No Automation Total

M SE M SE M SE M SE M SE M SE

Smart-alarm 10.53 (3.17) 36.65 (6.49) 23.59 (4.65) Smart-alarm 34.46 (6.79) 57.09 (5.93) 45.77 (5.82)
Rationalized 20.56 (7.37) 86.68 (11.74) 53.62 (6.24) Rationalized 56.36 (5.61) 80.91 (4.05) 68.64 (2.92)
Not Rationalized 19.75 (5.16) 31.86 (8.56) 25.81 (4.78) Not Rationalized 58.36 (5.80) 73.0 (4.50) 65.68 (4.15)
Total 16.94 (2.93) 51.73 (5.21) Total 49.73 (5.12) 70.33 (3.39)

Table 5 Table 7
Flare released (Mg) pairwise comparisons. Workload pairwise comparisons.
Condition Comparisons Mean Standard p Condition Comparisons Mean Standard p
Difference Error Difference Error

Alarm Alarm
Rationalized Non- 27.81 9.21 .04 Rationalized Non- 2.96 3 1
Alarm Rationalized Alarm Rationalized
Alarm Alarm
Smart Alarm 30.03 7.95 .011 Smart Alarm 22.86 4.27 .001
Smart Alarm Non- − 2.22 6.79 1 Smart Alarm Non- − 19.91 5.10 .009
Rationalized Rationalized
Alarm Alarm

Automation Trip No Trip − 34.79 6.81 <.001 Automation Trip No Trip − 20.61 4.42 .001
Application Application Application Application

on the system wherein the automation will shut down portions of the
process that are under normal operations to isolate the
out-of-control-process. Alarm management, on the other hand, is an
indirect solution that will only attempt to reduce the number of alarms
and present potentially relevant information to the operator. Therefore,
the safe-park automation may assist operator decision-making and
reduce operator mental processing time more effectively than alarm
management.
The alarm design results were mixed. First, in terms of perceived
workload, and regardless of automation, the smart alarm condition
resulted in the lowest reported workload scores – significantly lower
than both the rationalized and non-rationalized alarm conditions. In
terms of flare release, and again regardless of automation, the smart
alarm condition and non-rationalized alarm condition resulted in the
lowest amount of flare released (but were statistically similar to one
another). Upon inspection of the interaction between automation and
alarm conditions for the material lost through flare (Fig. 3.), we found
that the no-safe-park automation, rationalized alarm condition resulted

6
R.J. Simonson et al.
in the highest amount of material lost via flare release. At the same time,
the no-safe-park automation, smart alarm and the no-safe park,non-
rationalized alarm conditions were statistically insignificant from one
another – an inconsistency with prior investigations on the relationships
between rationalized alarm protocols and performance (Mahajan and
Surve, 2012; Adhitya et al., 2014). Additionally, comments from oper­
ators during the rationalized, no automation scenario indicated that
some operators had trouble finding certain alarms in question. This
suggests that the rationalized no-automation scenario may have been
more difficult to complete than other scenarios. In turn, this leads to the
possibility that material lost through flare may not have been affected by
alarm rationalization techniques, rather, only by automation schema.
Importantly, when considering both alarm rationalization and auto­
mation schemas as well as their effects on workload and material lost
through flare, the data indicates that smart alarming was the most
effective alarm state in this simulation. This is most likely due to the
significant concatenation in alarming that the smart alarm process
provides. Consistent with prior literature, appropriate alarm manage­
ment may provide operators with the tools needed to reduce their
workload and increase their performance. Subsequently, future research
should investigate the effects of varying task difficulty on material lost
through flare based on alarm rationalization techniques. This is most
likely due to the significant concatenation in alarming that the smart
alarm process provides. This result indicates that consistent with prior
literature, appropriate alarm management may provide operators with
the tools needed to reduce their workload and increase their perfor­
mance. Subsequently, future research should investigate the effects of
varying task difficulty on material lost through flare based on alarm
rationalization techniques.
Our results regarding the correlations between workload and per­
formance as measured by the amount of flare released are inconclusive.
The correlational values ranged from relatively low to relatively high.
We posit that these results may indicate that the underlying mechanisms
behind a task influence the relationship between workload and perfor­
mance, which may imply that similar assumptions from prior research
hold true (Cox-Fuenzalida, 2007). We found that a large dichotomy
between the correlations of workload and performance exists between
scenarios. Specifically, scenario 1 (rationalized alarm w/automation)
scenario 2 (rationalized alarm w/o automation) and scenario 4 (non-­
rationalized alarm w/automation) yielded the largest correlational
values while scenario 3 (smart alarm w/automation) scenario 5 (non-­
rationalized alarm w/o automation) and scenario 6 (smart alarm w/o
automation) yielded the lowest correlational values. Generally, sce­
narios with rationalized alarms or automation yielded the highest cor­
relations while scenarios without rationalization or automation yielded
the lowest except for scenario 3. However, without an experimental
design specifically geared towards understanding the causal effects of
alarm rationalization and automation on the ratio of workload and
performance levels, we are unable to determine if a true effect lies here.
Thus, future research should explore the relationships between the
correlations of workload and performance with alarm rationalization
schemas, automation, and their interactions from a causality point of
view. Implications for optimal levels of performance and workload may
yield a greater ratio between performance levels and workload over
longer-term periods of performance as compared to simply optimizing
alarm rationalization and automation techniques to yield the greatest
performance or workload level independently. Additionally, under­
standing the ratios of performance and workload on long-term outcomes
is required before conclusive evidence regarding these ratios is possible.
4.1. Limitations
Our findings that suggest the rationalized alarm system resulted in
the highest average flare released, however, this is likely a matter of the
scenarios used in this evaluation; the challenge of implementing the
simulated scenarios resulted in a few limitations. First, designing six
7
Applied Ergonomics 100 (2022) 103670
scenarios that were similar in complexity, yet involved different valves
(to avoid learning effects), proved to be somewhat difficult. As such,
some of the scenarios may have been more or less inherently difficult
based on the challenge the operators were required to find and solve.
The scenario in the rationalized alarm condition without automation
involved the failure of a pressure controller whereas all other scenarios
in this study involved valve failures that impacted vessel level. Addi­
tionally, this same scenario required two displays to garner the neces­
sary failure information whereas the other scenarios only required one
display. Hence, the differences in this scenario as compared to the other
five scenarios may have clouded the effect of the alarm design.
Additionally, due to the base scenario being a plant outage, many
possible problems were automatically eliminated. This automatic
elimination of problems may have reduced operator workload and, in
turn, the amount of time it took the operator to solve the challenges and
may have decreased variance between the conditions making it difficult
to detect performance differences. Further, the objective of our research
was to identify the performance and workload differences among non-
rationalized, rationalized, and smart alarming schemas with and
without automation. An inherent property of the alarm rationalization
schemas and automation is that the number of alarms that sound differs.
While the number of alarms is not the purpose of this investigation, they
are embedded within the alarming and automation conditions and thus
are a possible confound within this investigation. Other possible limi­
tations include those differences in alarm graphics (e.g., some alarm
graphics have relatively more information displayed on them than do
other alarm graphics) which may have introduced a confound that
resulted in increased workload or the amount of flare released. Addi­
tionally, our data resulted in collinearity and normality challenges in
some of our variables. Specifically, there was a lack of collinearity (r <
0.2) between workload and flare released for scenarios three and five.
We decided to include these variables in our analyses due to the
importance they have in our study. Finally, our limited sample size, due
to the in-situ and applied nature of this investigation, was not optimal
for our MANOVA. Future research should replicate this investigation to
determine if similar affects are found. Further, the use of our latin-
square condition order procedure required either 12 or 18 participants
to equally fulfill each randomized order. Subsequently, one order of
conditions only contained one participant instead of two.
5. Conclusion
The implications of this work provide insight into ways to improve
alarm rationalization and automation in regard to human performance
in chemical processing systems. First, this is the only study to date that
provides evidence that alarm strategies via rationalization, smart state-
based schemas, and automation provide a significant improvement in
performance and workload for chemical processing experts using a
human-in-the-loop, high fidelity simulation platform. Chemical pro­
cessing is complex, with modern equipment, product production, and
refinement techniques that have created environments where operators
are expected to oversee and manage multifaceted and complex tasks
with a multitude of interacting elements. Further, the product itself is
dangerous, particularly in fields such as ethylene development, with
mishandling potentially leading to hazardous scenarios. These scenarios
require full collaboration between operator and machine to stop errors
from propagating. While these modern systems are designed to achieve
longer and more stable periods of system homeostasis, the complexity of
the systems means that system instability can cause a chain reaction of
every monitor and alarm directly or possibly indirectly linked to it.
Through this mechanism, when an out-of-control process occurs, hun­
dreds of alarms may be produced as a result of one malfunction
(Strobhar, 2012). This requires that modern operators have the ability to
identify which alarms are and are not necessary in order to discover the
source of the malfunction.
Overall, the results of this investigation indicate that automation and
R.J. Simonson et al.
state-based alarm systems assisted operators to resolve system abnor­
malities, failures, or otherwise adverse process events. Improving
operator performance by these means provides organizations the ability
to enhance operators’ abilities to successfully safely and efficiently
complete their tasks. The rationalized alarm used in this study does not
match the expected results formed by way of prior evidence from pre­
vious research. Thus, further investigations into the effects of rational­
ized alarms on operator performance should be prioritized in future
research in this domain. Human-in-the-loop simulation-based research
is essential to fully understand the effects of alarm and automation
design in process control and other domains.
Funding
Financial support for this project was from the Center of Operator
Performance.
Declaration of competing interest
The authors declare the following financial interests/personal re­
lationships which may be considered as potential competing interests:
Financial support for this project was from the Center for Operator
Performance.
Acknowledgments
We would like to acknowledge Dr. David Strobhar of the Center for
Operator Performance for his contributions to this project, and his
consultation regarding alarm strategies and automation strategies in the
process-control industries.
References
Adhitya, A., Cheng, S.F., Lee, Z., Srinivasan, R., 2014. Quantifying the effectiveness of an
alarm management system through human factors studies. Comput. Chem. Eng. 67,
1–12.
Beebe, D., Ferrer, S., Logerot, D., 2013. The connection of peak alarm rates to plant
incidents and what you can do to minimize. Process Saf. Prog. 32 (1), 72–77.
Brown, N., 2003. Alarm management/The EEMUA guidelines in practice. Measurement
Control 36 (4), 114–119.
Cohen, J., 1988. Statistical Power Analysis for the Behavioral Sciences, second ed. L.
Erlbaum Associates, Hillsdale, N.J.
Corys, 2015. Hydrocarbons portal. https://www.corys.com/en/hydrocarbons-portal.
Cox-Fuenzalida, L.E., 2007. Effect of workload history on task performance. Hum.
Factors 49 (2), 277–291.
Dadashi, N., Golightly, D., Sharples, S., 2017. Seeing the woods for the trees: the problem
of information inefficiency and information overload on operator performance.
Cognit. Technol. Work 19 (4), 561–570.
de Visser, E., Parasuraman, R., 2011. Adaptive aiding of human-robot teaming: effects of
imperfect automation on performance, trust, and workload. J Cognitive Eng.
Decision Making 5 (2), 209–231.
Emam, E.A., 2015. Gas flaring in industry: an overview. Petroleum & coal 57 (5).
Endsley, M.R., 1987. The application of human factors to the development of expert
systems for advanced cockpits. In: Proceedings of the Human Factors Society Annual
Meeting, 31. SAGE Publications, Sage CA: Los Angeles, CA, pp. 1388–1392. No. 12.
Endsley, M.R., 1999. Level of automation effects on performance, situation awareness
and workload in a dynamic control task. Ergonomics 42 (3), 462–492.
Endsley, M.R., Kiris, E.O., 1995. The out-of-the-loop performance problem and level of
control in automation. Hum. Factors 37 (2), 381–394.
Fitts, P.M., 1951. Human Engineering for an Effective Air Navigation and Traffic Control
System. National Research Council, Washington, DC.
Goodwin, C.J., Goodwin, K.A., 2012. Methodological control in experimental research.
In: Research in Psychology: Methods and Design. Wiley., New Jersey, pp. 183–211.
Guerlain, S., Jamieson, G.A., Bullemer, P., Blair, R., 2002. The MPC elucidator: a case
study in the design for human-automation interaction. IEEE Trans. Syst. Man
Cybern. Syst. Hum. 32 (1), 25–40.
Applied Ergonomics 100 (2022) 103670
Hancock, P.A., Jagacinski, R.J., Parasuraman, R., Wickens, C.D., Wilson, G.F., Kaber, D.
B., 2013. Human-automation interaction research: past, present, and future. Ergon.
Des 21 (2), 9–14.
Hart, S.G., 1986. NASA Task Load Index (TLX). Volume 1.0. Paper and pencil package.
https://humansystems.arc.nasa.gov/groups/TLX/downloads/TLXScale.pdf.
Hart, S.G., Staveland, L.E., 1988. Development of NASA-TLX (task load index): results of
empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (Eds.), Human
Mental Workload. North Holland Press., Amsterdam.
Hollender, M., Beuthel, C., 2007. Intelligent alarming. ABB Rev. 1, 20–23.
Hollifield, B., 2010. Understanding and Applying the ANSI/ISA 18.2 Alarm Management
Standard. PAS Inc., Houston.
Kaber, D.B., Endsley, M.R., 2004. The effects of level of automation and adaptive
automation on human performance, situation awareness and workload in a dynamic
control task. Theor. Issues Ergon. Sci. 5 (2), 113–153.
Lee, J.D., Gore, B.F., Campbell, J.L., 1999. Display alternatives for in-vehicle warning
and sign information: message style, location, and modality. Transport. Hum.
Factors 1 (4), 347–375.
Lee, J.D., Seppelt, B.D., 2009. Human factors in automation design. In: Shimon, Y.N.
(Ed.), Springer Handbook of Automation. Springer., Berlin, pp. 417–436.
Lee, J.D., Seppelt, B.D., 2006. Human factors and ergonomics in automation design. In:
Salvendy, G. (Ed.), Handbook of Human Factors and Ergonomics. Wiley., New York,
pp. 1615–1642.
Lorenz, B., Di Nocera, F., Röttger, S., Parasuraman, R., 2001. The effects of level of
automation on the out-of-the-loop unfamiliarity in a complex dynamic fault-
management task during simulated spaceflight operations. In: Proceedings of the
Human Factors and Ergonomics Society Annual Meeting, 45. SAGE Publications,
Sage CA: Los Angeles, CA, pp. 44–48. No. 2.
Mahajan, S.A., Surve, V.G., 2012. Operator performance enhancement by alarm
management. In: SPE International Production and Operations Conference &
Exhibition. Society of Petroleum Engineers.
Miller, G.A., 1956. The magical number seven, plus or minus two: some limits on our
capacity for processing information. Psychol. Rev. 63 (2), 81.
NIST/SEMATECH e-Handbook of Statistical Methods, 2013. 6.3.2. What are variables
control charts? https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc32.
htm#Trend.
Nochur, A., Vedam, H., Koene, J., 2001. Alarm performance metrics. IFAC Proc. Vol. 34
(27), 203–208.
Ohio, E.P.A., 2014. Understanding the basics of gas flaring. http://epa.ohio.gov/Portals
/27/oilandgas/BasicsofGasFlaring.pdf.
Parasuraman, R., Sheridan, T.B., Wickens, C.D., 2000. A model for types and levels of
human interaction with automation. IEEE Trans. Syst. Man Cybern. Syst. Hum. 30
(3), 286–297.
Popken, A., Krems, J.F., 2017. Automation and situation awareness. In: Boy, G.A. (Ed.),
The Handbook of Human-Machine Interaction. CRC Press., Boca Raton, FL,
pp. 75–90.
Reddy, Y.J., Mehta, B.R., 2015. Alarm management systems. In: Mehta, B.R., Reddy, Y.J.
(Eds.), Industrial Process Automation Systems: Design and Implementation.
Elsevier., Amsterdam, pp. 569–582. https://doi.org/10.1016/B978-0-12-800939-
0.00021-8.
Riley, V., 1989. A general model of mixed-initiative human-machine systems. In:
Proceedings of the Human Factors Society Annual Meeting, 33. SAGE Publications,
Sage CA: Los Angeles, CA, pp. 124–128. No. 2.
Rockwell Automation, 2017a. Alarm Management and Implementation. In: https://liter
ature.rockwellautomation.com/idc/groups/literature/documents/wp/proce
s-wp015_-en-p.pdf.
Rockwell Automation, 2017b. Economic and Effective Alarm Management. In: htt
ps://literature.rockwellautomation.com/idc/groups/literature/documents/wp/pro
ces-wp013_-en-p.pdf.
Sauer, J., Kao, C.S., Wastell, D., Nickel, P., 2011. Explicit control of adaptive automation
under different levels of environmental stress. Ergonomics 54 (8), 755–766.
Siemens, 2018. Alarm Management in Simatic Pcs 7 – Smart Alarm Hiding. Chicago, IL.
https://cache.industry.siemens.com/dl/files/984/55699984/att_966468
/v2/55699984_Alarm_Hiding_DOC_PCS7_V90SP1_en.pdf.
Sheridan, T.B., Verplank, W.L., 1978. Human and Computer Control of Undersea
Teleoperators. Massachusetts Inst of Tech Cambridge Man-Machine Systems Lab.
https://apps.dtic.mil/dtic/tr/fulltext/u2/a057655.pdf.
Stanton, N.A., Salmon, P., Jenkins, D., Walker, G., 2009. Human Factors in the Design
and Evaluation of Central Control Room Operations. CRC Press., Boca Raton, FL.
Strobhar, D.A., 2012. Human Factors in Process Plant Operation. Momentum Press,
pp. 3–5.
Veland, O., Kaarstad, M., Seim, L.A., Foerdestroemmen, N.T., 2001. Useful and Useable
Alarm Systems: Recommended Properties (No. HWR–679). Institutt for
Energiteknikk.