Reliability Engineering and System Safety 227 (2022) 108727

Contents lists available at ScienceDirect

Reliability Engineering and System Safety

journal homepage: www.elsevier.com/locate/ress

Empirical study on human error probability of

procedure-extraneous behaviors
Yochan Kim a, *, Sun Yeong Choi a, Jinkyun Park a, Jaewhan Kim a
a
Risk Assessment Research Team, Korea Atomic Energy Research Institute (KAERI), 111 Daedeok-daero 989 Beon-gil, Yuseong-gu, Daejeon 34057, Republic of Korea

A R T I C L E I N F O A B S T R A C T

Keywords: To realistically quantify human reliability in socio-technical systems, it is useful to obtain statistical evidence
Full-scope simulator from empirical data. A procedure-extraneous error, which is a type of commission error, can occur when an
Human error probability operator attempts to operate a system or component that is not guided in the relevant procedure. Because these
Human reliability analysis
behaviors are rarely found in system environments in which the procedures are strictly followed, they have not
HuREX framework
Poisson regression
been sufficiently studied from a quantitative point of view. This study aims to estimate the occurrence proba­
Procedure-extraneous behaviors bility of procedure-extraneous behaviors based on data from a full-scope simulator of a nuclear power plant. For
this analysis, 107 simulation records of the APR1400 plant and human errors identified by experts in various
fields were analyzed by Poisson regression. Using the resulting coefficient estimates from the regression model,
the expected recovery probabilities, and the performance time required for human events, the occurrence
probability of procedure-extraneous behavior was predicted. Depending on the situation and recoverability,
various values of human error probability were derived between 1 and 10− 6. Limitations from data scarcity and
the implications of the results are discussed.

1. Introduction
Because human errors are known to contribute to both the reliability
and the risk of socio-technical systems [1–5], such as nuclear power
plants, railway engineering, and maritime transportation, various kinds
of models and techniques have been developed to assess potential
human failures in the operation of such systems. With the development
of human reliability analysis (HRA) methods, many researchers have
collected and analyzed data that can provide empirical evidence for a
quantitative basis of the methods [5–8]. These efforts have not only
facilitated more realistic assessments of human errors, but have also
provided a base for examining the error mechanisms that had previously
been overlooked.
However, empirical studies on human behaviors that aggravate
system safety or potentially exacerbate accident situations have not been
sufficiently conducted thus far. Some literature refers to such behavior
as a commission error [9–12], which has been defined in the nuclear
field as “incorrectly performing a system-required task or action, or
performing an extraneous task that is not required and might lead to
worsening the accident progression or cause an initiating event” [12]. In
many operating environments based on procedures, these commission
errors can be distinguished into two types: (1) the incorrect performance
of a procedural task, and (2) procedure-extraneous human errors.
Incorrect performance of the operating procedure includes cases in
which the operator follows the wrong procedure or step due to a
misdiagnosis of the situation, as well as cases in which the operator’s
device selection, manipulation orientation, or control quantity is not
appropriate when operating some equipment according to the proper
procedure. A procedure-extraneous behavior includes cases in which the
operator attempts to operate a system or component that is not guided in
the procedure. Examples include operating a device by an accidental slip
or involuntary action, taking proactive or habitual actions based on the
operator’s knowledge (rather than the procedure), and activating a de­
vice to check its operability.
Commission error is often not quantitatively modeled in many HRA
applications due to the complex identification process of the potential
errors, insufficient empirical data on commission error occurrences, and
the demands for a sophisticated modeling of their consequences [11].
The following examples highlight some efforts to quantitatively estimate
commission error probabilities. The ATHEANA method allows analysts
to identify commission errors, but lacks standardized identification rules
or data for the quantification [13]. CESA-Q is a methodology developed

* Corresponding author.
E-mail addresses: yochankim@kaeri.re.kr (Y. Kim), sychoi@kaeri.re.kr (S.Y. Choi), kshpjk@kaeri.re.kr (J. Park), jhkim4@kaeri.re.kr (J. Kim).

https://doi.org/10.1016/j.ress.2022.108727
Received 24 September 2021; Received in revised form 11 July 2022; Accepted 20 July 2022
Available online 21 July 2022
0951-8320/© 2022 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
Y. Kim et al.
for commission error analysis and mainly predicts the probability of
incorrect performance that may occur during decision-making based on
procedures [14]. Podofillini et al. presented a quantification model of
the CESA-Q method based on data derived from 26 operational events
[15]. Another approach, MDTA, only deals with the commission errors
related to the procedures or step selection [16]. The error probability of
the selected errors is quantified based on thermal-hydraulic analysis,
cause-based decision trees, and common cause failures.
Some empirical studies on the probability of commission errors
related to an incorrect performance in following a procedure have
recently been published. For example, Jung et al. estimated human error
probabilities (HEPs) for incorrect information verification, incorrect
selection of the procedure/step, and incorrect device operation based on
simulator data through the HuREX system [6]. Massaiu and Fernandes
calculated the commission error probabilities of basic identification
tasks through micro-task experiments [17]. On the other hand, the error
probability of procedure-extraneous behaviors has not been empirically
investigated, and has often been considered negligible in many HRA
applications.
This study aims to estimate the HEP due to procedure-extraneous
behaviors, which to date has not been studied using empirical data,
based on data from a full-scope simulator of a nuclear power plant. For
this purpose, the HuREX data extracted from an APR1400 full-scope
simulator was analyzed. The human reliability characteristics of
procedure-extraneous behaviors were identified through observations of
the behavior of licensed crews in the simulators, and the probability of
occurrence was quantitatively estimated based on the characteristics.
This study scrutinized non-procedural human behaviors in emergency
situations, which are generally responded to by strictly following the
corresponding procedures. In other words, the operator actions analyzed
in this work do not imply knowledge-based behaviors in situations
where a procedural guide is absent or wholly improper. A full descrip­
tion of the error probability estimation employed here is available in a
technical report [18].
2. Collected data
For analyzing procedure-extraneous behaviors, the records from a
full-scope simulator of the APR1400 nuclear power plant were collected
from the period 2017 to 2019 [7]. The participants in the simulations
were licensed operators conducting regular simulation training. Half of
the operators were relatively more experienced because they were
working in commercially operating plants at the time of record gath­
ering, while the rest were working in a plant unit under trial operation
and therefore had relatively less experience. The APR1400 is equipped
with a control room where all human–machine interfaces are comput­
erized. The crews operate the power plant using digital displays, soft
controls, and computer-based procedures (CBPs). A total of 107 simu­
lations for 12 emergency scenarios were recorded; Table 1 summarizes
the numbers of simulation runs and the average period of simulation for
each scenario.
The records acquired from the simulator include videos showing the
act of manipulating the displays, operation history logs of the devices,
simulated plant parameter logs, injected malfunction input logs for the
scenarios, and questionnaire data about the operators’ experience. Ac­
cording to the definition of an unsafe act as defined in the HuREX
framework, the data collectors identified all the behaviors that deviated
from the procedure (such as a system or component manipulation not
specified in the procedure), and if a behavior had a negative conse­
quence on plant safety, it was considered as a human error [6]. In other
words, via consultations with plant experts and HRA experts, only the
device operations made outside the procedures that negatively affected
the safety of the power plant were determined to be
procedure-extraneous human errors.
2
Reliability Engineering and System Safety 227 (2022) 108727
Table 1
Simulation runs and average simulation time for each scenario.
Scenario Simulation Average simulation
runs time (min)
LOCA (loss of coolant accident) from a pilot- 8 19.338
operated safety relief valve (POSRV)
LOCA from direct vessel injection (DVI) 9 19.589
LOCA from a letdown valve 8 26.373
LOCA from a reactor coolant pump (RCP) 8 17.369
seal
LOCA from a low-temperature overpressure 12 24.238
protection (LTOP) valve
LOCA with safety injection (SI) failure 9 14.172
SGTR (steam generator tube rupture) 7 24.214
SGTR with a failure of the CBP 11 24.156
SGTR with failures of 4 radiation indicators 11 26.647
SGTR with SI failure 8 14.890
SBO (station blackout) 6 23.617
LOAF (loss of all feedwater) requiring feed 10 27.345
and bleed operation (F&B)
3. Data analysis
3.1. Qualitative characteristics
By observing the behaviors of the operators in the simulator, we
identified the commission errors and their characteristics. The operators
sometimes exhibited procedure-extraneous behaviors during the emer­
gency simulations. In many cases, the behaviors were proactive re­
sponses made by anticipating the contents of the given procedure that
had not yet been entered, re-manipulations to confirm previously
operated devices, or closing additional valves in the isolated systems.
These actions were not distinguished as commission errors in this study
because they did not affect plant safety. However, some actions were
judged to have a negative impact on plant safety and were thus classified
as procedure-extraneous human errors. In total, 14 procedure-
extraneous human errors were observed during the 107 simulations.
Table 2 briefly summarizes the contexts of the observed procedure-
extraneous errors. Because these errors occurred somewhat rarely,
they were not observed in certain scenarios such as SBO and LOCA from
a letdown valve.
The characteristics of the 14 human errors are summarized as fol­
lows. First, all procedure-extraneous errors occurred while the operators
were conducting actions consciously responding to the accident situa­
tion; i.e., there were no commission errors related to accidental device
operations such as a simple hand slip or an unconscious pressing of a
button. The errors occurred when the operator’s situational assessment
was not appropriate or their responsive plan was inappropriate while
trying to manipulate a particular component based on memory. Second,
among the 14 human errors, 2 errors were identified as actions that
significantly influenced the critical safety functions of the power plant
(IDs 5 and 7 in Table 2). The remaining 12 commission errors also
changed the state of the components, but had no direct effect on the
safety functions of the power plant. Third, it was found that 11 out of the
14 human errors were committed by the operators with relatively less
experience (i.e., crews from a plant under trial operation), including the
2 human errors that directly affected the safety functions. Fourth, 11
human errors were initiated by the supervisor’s instructions and ap­
provals, while the other 3 were caused by a turbine operator’s own at­
tempts to operate the components. Lastly, among all human errors, only
1 was recovered (ID 7 in Table 2). For the other human errors, the
simulations were terminated before their recoveries (see Sect. 3.3), and
thus their potential recoveries could not be observed.
In particular, the fifth commission error in Table 2 is judged to have a
meaningful effect from the characteristics of the accident scenario. If a
leak occurs in the POSRV valve, the PRZ pressure is lowered, and SI is
actuated. At this time, since the PRZ level exceeds its normal range due
Y. Kim et al. Reliability Engineering and System Safety 227 (2022) 108727

Table 2
Identified procedure-extraneous human errors from the APR1400 simulator.
ID Scenario Description Crew Error Safety
experience initiator function

1 LOCA from an RCP seal After the main steam line was isolated, an operator recommended to cool the RCS* using a Less Supervisor Not
main steam isolation valve. The supervisor instructed to open the isolation valve and the experienced affected
operator opened it. (The procedure guides to open an isolation bypass valve, not the
isolation valve.)
2 LOCA from an RCP seal Similar error as ID 1 More Supervisor Not
experienced affected
3/ LOCA from DVI The same error as ID 1. Two commission errors were observed during this simulation. Less Turbine Not
4 experienced operator affected
5 LOCA from a POSRV An SI pump had been started and the termination condition of SI was not satisfied at that Less Supervisor Affected
point. But the supervisor ordered to stop the pump because of the high level of the PZR**. experienced
The operator stopped the SI pump.
6 LOAF requiring F&B The supervisor proactively directed to initiate SI because of a lowering level of the PZR. The Less Supervisor Not
operator pressed the SI actuation signal button. However, SI was not injected owing to the experienced affected
high pressure of the PZR.
7 SGTR An operator recommended to open the atmospheric dump valves to decrease the pressure of Less Turbine Affected
the sound SG*** before the ingress of the relevant procedure. He then opened the dump experienced operator
valves of the ruptured SG.
8 SGTR Similar error as ID 1 More Supervisor Not
experienced affected
9 SGTR with the failure of 4 Similar error as ID 1 Less Supervisor Not
radiation indicators experienced affected
10 SGTR with the failure of 4 Similar error as ID 1 More Supervisor Not
radiation indicators experienced affected
11 SGTR with CBP failure Similar error as ID 1 Less Supervisor Not
experienced affected
12 SGTR with CBP failure Similar error as ID 1 Less Supervisor Not
experienced affected
13 SGTR with CBP failure Similar error as ID 1 Less Supervisor Not
experienced affected
14 SGTR with CBP failure As the PZR level decreased, an operator recommended to stop two RCPs in loop A. After Less Supervisor Not
receiving approval from the supervisor, he stopped the pumps. (The procedure instructs to experienced affected
stop one RCP per loop, not two pumps in the same loop.)
*
RCS: reactor coolant system.
**
PZR: pressurizer.
***
SG: steam generator.

to the SI, the purpose of controlling the PRZ level to stay within the
normal range and that of satisfying the SI termination condition are in
conflict. It was judged that this special condition with such a leakage
increased the complexity of the operator’s decision-making. The
complexity of this accident could have the effect of increasing the
likelihood of commission errors by operators with little experience.
3.2. Regression analysis and result
3.2.1. Technique and variables
A regression analysis was performed to estimate the error probability
of the observed procedure-extraneous behaviors. Since the procedure-
extraneous behaviors were not conducted at times when a specific
task was required, it was difficult to estimate the ratio between the
human error frequency and the task performance frequency (or, the
error opportunity) like other kinds of HEPs. Instead, in this study, the
occurrence rate of procedure-extraneous errors per unit time was esti­
mated. It was assumed that the procedure-extraneous errors follow a
Poisson distribution, which implies that individual errors are indepen­
dent of each other. Poisson regression is often used to estimate the
occurrence rate of an event within a given time or space [19]. Expressing
this as a formula takes the following form:
α + ̂βx,
ln(E(Y|x)) = ln(obs) + ̂
wherê α is the intercept coefficient, ̂
β is the regression coefficient for the
context variable x, and Y is the number of events. Obs is an offset vari­
able indicating the unit time of observation or exposure.
exponent of the intercept coefficient [i.e., exp(̂α )] can be regarded as an
estimate of the nominal error rate, while the exponent of the regression
coefficient [i.e., exp(̂
β)] can be seen as the effect of the context variable
when the nominal condition of the context variable is expressed by x =
0.
Dependent and independent variables were established based on the
qualitative characteristics discussed in Section 3.1 to statistically
analyze the procedure-extraneous errors. For the dependent variable of
this regression model, Y, the procedure-extraneous errors affecting the
critical safety functions were used. For the independent variables x, two
context variables, namely conflict in operational purpose and crew
experience, were considered based on the qualitative error analysis.
Conflict in operational purpose refers to the scenario property shown in
the LOCA from POSRV scenario, and crew experience refers to whether
the operators participating in the simulations had commercial operation
experience with the same type of plant as in the simulator. Because
several kinds of combinational effects are imaginable from the two
variables, statistical models generated by different combinations
including the individual variables and the interaction variables were
evaluated and compared, as detailed in the next section. Lastly, the
simulation time was used as the offset variable.
In order to select the significant variables and measure the goodness-
of-fit of the regression model, the Bayesian information criterion (BIC)
(1) was used. BIC evaluates the efficiency and prediction accuracy of a
regression model by Eq. (2):
∏
EP = (4.397E − 04) ∗ 25.891CC ∗T ∗ RCi (2)
i

Because the above formula has a log-linear form, a multiplicative

where ̂
L is the maximized value of the likelihood function of the
expression representing the relation between an error rate and a context
regression model, k is the number of parameters, and n is the number of
variable can be obtained when both sides are exponential. That is, the

3
Y. Kim et al. Reliability Engineering and System Safety 227 (2022) 108727

data samples. A smaller BIC value means a better-fitted model. ∏

A likelihood ratio test was also performed to verify whether indi­ EP = (4.397E − 04) ∗ 25.891CC ∗T ∗ RCi (3)
vidual models fit the data more significantly than a null model. This test
i

compares the difference between the residual deviance of the model
including selected variables and the residual deviance of the null model
based on chi-square distribution. If the p-value is less than a pre­
determined level (e.g., 0.05), the model is said to be statistically
significant.
4. Result
We estimated the occurrence rate per minute of procedure-
extraneous errors using seven different models. Table 3 summarizes
the estimates of the regression models and their BIC values. A dagger in
the last column means that the model is statistically significant at the 5%
significance level in terms of the likelihood ratio test. The BIC values and
the likelihood ratio test in this table show that the sixth model is best
suited for predicting the procedure-extraneous error rates. This model
includes an interaction variable between the crew experience and con­
flict in operational purpose variables. When the crew has insufficient
experience in commercial operation and the accident scenario involves
any operational goal conflicts, the occurrence rate of the procedure-
extraneous errors can increase. With the sixth model, the nominal
occurrence rate was estimated to be 4.397E-04/min, and the effect of the
interaction variable was 25.891.
4.1. Application of the result to HRA
To implement the meanings of the regression estimates into the HEP
estimation of procedure-extraneous behaviors, the estimated regression
coefficients can substitute for the coefficients in Eq. (1). In addition,
since the occurrence rate of the procedure-extraneous behaviors was
estimated as the occurrence probability per unit time in the previous
regression analysis, it should be multiplied by a meaningful operator
performance time. Lastly, because several kinds of recoveries for the
procedure-extraneous errors are expected, the recovery possibilities
should be added in the HEP calculation. Constructed based on these
considerations, Eq. (3) gives the formula for predicting the error prob­
ability of a procedure-extraneous behavior:
where EP is the error probability of a procedure-extraneous behavior, CC
is a binary variable indicating whether the crew experience is insuffi­
cient and the accident scenario includes any conflicts in operational
purpose, T is the human performance time (unit: min), and RCi is the
recovery failure probability.
First of all, because the estimates derived from the regression anal­
ysis are applied to the rate of occurrence per minute, it is important to
apply a meaningful human performance time, T, for human failure
events. Human reliability analysis mainly analyzes the time available for
the completion of the given event and the time required to perform the
tasks of the event. Here, we selected the time required as the significant
performance time for this application for two reasons. First, in the pre­
vious qualitative analysis of the procedure-extraneous errors, we found
that all the errors were committed while operators were coping with
actual accident situations, meaning that there were no commission er­
rors irrelevant to the accident situation. Therefore, it is reasonable to
consider the occurrence probability of procedure-extraneous errors in
terms of the time actually required to perform the tasks for the given
human failure event. The second reason is that the time available is often
shortened in unstable situations in power plants, while it is conversely
lengthened when the plant situation is stable. If the time available is
multiplied by the regression coefficient instead of the time required, the
counterintuitive result is that the more stable the accident situation, the
higher the probability of procedure-extraneous errors. In this study,
referring to data regarding the time required for APR1400 events pub­
lished in a report [20], it was assumed that the human performance time
is distributed between approximately 3 min and 60 min.
Additionally, it was assumed that the following three types of re­
coveries can be considered: (1) self-recovery of the crew, (2) critical
safety function monitoring by a shift technical advisor, and (3) recheck
by a shift change. The crew’s self-recovery is a case in which an operator
discovers and corrects the commission error by themself. Its failure
probability was assumed to be 0.5 because one of the two critical errors
was recovered within 2 min in the HuREX data. Second, since the shift
technical advisor of a nuclear power plant usually monitors the critical
safety functions every 15 min in any emergency situation, it was

Table 3
Coefficient estimates and BIC values of various Poisson regression models.
Model Variable Mean Standard Exp (Mean) BIC
estimate error

(1) Error: Crew experience (intercept) –24.33 3384.97 2.707E-11 26.683

Crew experience: Less 17.96 3384.97 6.285E+07
(2) Error: Conflict in operational purpose (intercept) –7.699 1.000 4.531E-04 26.659
Conflict in operational purpose: Exist 2.658 1.414 1.427E+01
(3) Error: Crew experience + Conflict in operational purpose (intercept) –25.667 5147.768 7.128E-12 28.784
Crew experience: Less 18.675 5147.768 1.290E+08
Conflict in operational purpose: Exist 2.517 1.414 1.239E+01
(4) Error: Crew experience + (Crew experience & Conflict in operational (intercept) –25.333 5580.878 9.957E-12 28.784
purpose) Crew experience: Less 18.341 5580.878 9.231E+07
Crew experience: Less & Conflict in 2.517 1.414 1.239E+01
operational purpose: Exist
(5) Error: Conflict in operational purpose + (Crew experience & Conflict (intercept) –7.699 1.000 4.531E-04 30.199
in operational purpose) Conflict in operational purpose: Exist –13.694 3278.782 1.130E-06
Crew experience: Less & Conflict in 16.918 3278.782 2.225E+07
operational purpose: Exist
(6) Error: (Crew experience & Conflict in operational purpose) (intercept) –7.729 1.000 4.397E-04 25.586†
Crew experience: Less & Conflict in 3.254 1.414 25.891
operational purpose: Exist
(7) Error: Crew experience + Conflict in operational purpose + (Crew (intercept) –2.533E+01 5.735E+03 9.994E-12 33.457
experience & Conflict in operational purpose) Crew experience: Less 1.834E+01 5.735E+03 9.198E+07
Conflict in operational purpose: Exist –6.407E-02 2.490E+04 9.379E-01
Crew experience: Less & Conflict in 2.581 2.490E+04 1.321E+01
operational purpose: Exist

† p<0.05.

4
Y. Kim et al. Reliability Engineering and System Safety 227 (2022) 108727

assumed that this kind of recovery can be attempted with a failure Table 4
probability of 0.14 when there is a time margin of about 45 min. Here, Applied results of the regression coefficients for calculating procedure-
the assumed failure probability of 0.14 implies an approximate proba­ extraneous error probability.
bility of the conditional failure of a successor action when the ante­ T CC RC (Self- RC (STA) RC (shift change) EP
cedent action and successor action have an intermediate level of (min) review)* ** ***
dependence, which was suggested by Swain and Guttman [21]. Finally, 3 TRUE TRUE TRUE TRUE 1.20E-
we examined the possibility that a human error can be recovered by a 04
new crew shift, as nuclear power plant crews are rotated every 8 h. It 3 TRUE TRUE TRUE FALSE 2.39E-
03
was assumed that the new crew had little dependency on the previous
3 TRUE TRUE FALSE TRUE 8.54E-
crew, so the recovery failure probability was assumed to be 0.05 [21]. 04
Table 4 shows the procedure-extraneous error probability in various 3 TRUE TRUE FALSE FALSE 1.71E-
situations by combining the above assumptions. The error probability 02
ranged from 4.62E-06 to 6.83E-01. When the operator had commercial 3 TRUE FALSE TRUE TRUE 2.39E-
04
experience or there was no operational conflict in the scenario, the error 3 TRUE FALSE TRUE FALSE 4.78E-
probability was significantly lowered. Therefore, when this study is 03
applied to the HRA of a plant in commercial operation, the procedure- 3 TRUE FALSE FALSE TRUE 1.71E-
extraneous error probability is expected to be between 4.62E-06 and 03
3 TRUE FALSE FALSE FALSE 3.42E-
2.64E-02. From this table, it is revealed that the length of the human
02
performance time is also a very important factor in estimating error 3 FALSE TRUE TRUE TRUE 4.62E-
probability. Considering that the recoveries by shift change and shift 06
technical advisor are also closely related to the time margin, it can be 3 FALSE TRUE TRUE FALSE 9.23E-
seen that the error probability is greatly affected by time factors. 05
3 FALSE TRUE FALSE TRUE 3.30E-
05
5. Discussion and conclusion 3 FALSE TRUE FALSE FALSE 6.60E-
04
5.1. Discussion 3 FALSE FALSE TRUE TRUE 9.23E-
06
3 FALSE FALSE TRUE FALSE 1.85E-
This study presented the process and the results of a statistical 04
analysis to calculate the occurrence probability of procedure-extraneous 3 FALSE FALSE FALSE TRUE 6.60E-
human errors, a type of commission error. This study is significant in 05
that it provides a mathematical basis for predicting human reliability as 3 FALSE FALSE FALSE FALSE 1.32E-
03
part of a novel empirical data analysis approach to procedure- 60 TRUE TRUE TRUE TRUE 2.39E-
extraneous human errors. High-quality data generated through peer 03
reviews of data analysts was collected and analyzed to predict the 60 TRUE TRUE TRUE FALSE 4.78E-
relationship between contextual factors and human error probability. 02
60 TRUE TRUE FALSE TRUE 1.71E-
For this empirical study, 107 simulations (i.e., 2362 min in total) were
02
analyzed by experts in various fields. 60 TRUE TRUE FALSE FALSE 3.42E-
The obtained results have several important implications, as follows. 01
First, the results showed that significant procedure-extraneous human 60 TRUE FALSE TRUE TRUE 4.78E-
errors can occur in some cases, and that certain measures can appro­ 03
60 TRUE FALSE TRUE FALSE 9.56E-
priately reduce their occurrence probability. Sufficient operational 02
experience, crew training, and the establishment of procedural strate­ 60 TRUE FALSE FALSE TRUE 3.42E-
gies that can minimize conflicts of operational purpose in accident sit­ 02
uations are beneficial in reducing procedure-extraneous errors. In 60 TRUE FALSE FALSE FALSE 6.83E-
01
addition, (1) proper human–machine interface design for checking the
60 FALSE TRUE TRUE TRUE 9.23E-
safety critical functions, (2) periodic monitoring of the functions, and (3) 05
securing sufficient time available all enable the recoveries of these kinds 60 FALSE TRUE TRUE FALSE 1.85E-
of human errors, which are significant for the safety of nuclear power 03
plants. 60 FALSE TRUE FALSE TRUE 6.60E-
04
From the application of the regression coefficients, it was revealed 60 FALSE TRUE FALSE FALSE 1.32E-
that the error probability of the procedure-extraneous behaviors can be 02
predicted as various values between 10− 2 and 10− 6 depending on the 60 FALSE FALSE TRUE TRUE 1.85E-
performance time and the status of the contextual factors. Park et al. [1] 04
60 FALSE FALSE TRUE FALSE 3.69E-
and Kim et al. [7] estimated that the human error probabilities of
03
cognitive tasks in digital main control rooms were between 10− 1 and 60 FALSE FALSE FALSE TRUE 1.32E-
10− 5. This means that procedure-extraneous human errors can occur at a 03
negligible level in some cases but can significantly affect human reli­ 60 FALSE FALSE FALSE FALSE 2.64E-
ability in others. As mentioned earlier, when a relatively inexperienced 02

operator faces a conflicting situation, this type of commission error can *RC (Self-review): Self-recovery of the crew; ** RC (STA): Critical safety func­
have a non-trivial impact on system safety based on the time margin. tion monitoring by a shift technical advisor; *** RC (shift change): Recheck by a
Even if the probability of procedure-extraneous human errors is not shift change.
modeled in HRA using the results of this study, the derived nominal
error probability can be used as a meaningful source to derive the
minimum error probability of a human failure event. For example,
Table 4 shows that the error probability was predicted to be at least
4.62E-06 when the time margin is sufficiently long and the time required

5
Y. Kim et al.
is short. Like in some studies to determine the minimum bound of an
individual HEP [22–24], the minimum values of this study can likewise
be considered as a form of evidence supporting the minimum bounds.
We note that there is a large uncertainty in the results due to the
small number of data points. Since only two human errors were
considered for the regression analysis, the variables were selected sub­
jectively based on a causality investigation of the errors. The HuREX
framework allows the gathering of the values of various variables, for
which statistical fitness as well as expert knowledge is useful in order to
select meaningful variables [25]. Since sufficient data for statistical
analysis was not secured in this study, the variables were selected based
solely on expert knowledge. The subjectivity of this selection could be
validated with further data analysis.
Because the number of error data was insufficient, the statistical
significance of the regression coefficients could also be weak. Moreover,
since the APR1400 has only recently started commercial operation, the
overall experience levels of the APR1400 plant crews were relatively low
compared to those in other plants. For generating more robust insights
into human errors, collecting more data from various environments is
required.
Another limitation is that the recovery failure probabilities in the
application study were not based on empirical data. It is necessary to
discuss not only the lack of recovery data but also the appropriateness of
the dependency rule assumed in this study. There has been a lot of
discussion on how to quantify dependency [26], and recently, important
definitions and quantification approaches have been suggested [27–29].
In addition to appropriate data treatment for dependencies, new tech­
nologies for dependency can be utilized to obtain more realistic results
in the future.
5.2. Future work
This statistical study was performed for the general probability of
procedure-extraneous human errors. In the future, it is important to
identify the plant-specific mechanisms of the procedure-extraneous er­
rors and to model how these mechanisms can aggravate the condition of
the power plant. In addition to the previously developed methods such
as CESA-Q and MDTA, it is necessary to develop a method that can
scrutinize the mechanisms of procedure-extraneous behaviors.
CRediT authorship contribution statement
Yochan Kim: Writing – review & editing, Writing – original draft,
Methodology, Data curation, Conceptualization. Sun Yeong Choi:
Investigation, Data curation. Jinkyun Park: Supervision, Project
administration, Investigation. Jaewhan Kim: Supervision, Investiga­
tion, Conceptualization.
Declaration of Competing Interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
Data Availability
The authors do not have permission to share data.
Acknowledgement
This work was supported by the Nuclear Safety Research Program
through the Korea Foundation Of Nuclear Safety (KoFONS) using a
financial resource granted by the Nuclear Safety and Security Commis­
sion (NSSC) of the Republic of Korea (No. 2101054). This work was also
partially supported by the HRA for Digital I&C – KAERI project, funded
6
Reliability Engineering and System Safety 227 (2022) 108727
by the Electric Power Research Institute (EPRI Agreement No.:
10011321). We thank Prof. Kim in Chosun University, Prof. Lee in Ulsan
National Institute of Science and Technology, Mr. Song in Korea Hydro
& Nuclear Power Company, and Dr. Jung and Dr. Kim in Korea Atomic
Energy Research Institute for their advice on human error identification.
References
[1] Park J, Boring RL, Ulrich TA, Lew R, Lee S, Park B, Kim J. A framework to collect
human reliability analysis data for nuclear power plants using a simplified
simulator and student operators. Reliab Eng Syst Saf 2022;221:108326.
[2] Zarei E, Khan F, Abbassi R. Importance of human reliability in process operation: a
critical analysis. Reliab Eng Syst Saf 2021;211:107607.
[3] Catelani M, Ciani L, Guidi G, Patrizi G. An enhanced SHERPA (E-SHERPA) method
for human reliability analysis in railway engineering. Reliab Eng Syst Saf 2021;
215:107866.
[4] Wu B, Yip TL, Yan X, Soares CG. Review of techniques and challenges of human
and organizational factors analysis in maritime transportation. Reliab Eng Syst Saf
2022;219:108249.
[5] Riccardo P, Marilia R, Paltrinieri N, Massaiu S, Costantino F, Di Gravio G,
Boring RL. Human reliability analysis: exploring the intellectual structure of a
research field. Reliab Eng Syst Saf 2020:107102.
[6] Jung W, Park J, Kim Y, Choi SY, Kim S. HuREX–A framework of HRA data
collection from simulators in nuclear power plants. Reliab Eng Syst Saf 2020;194:
106235.
[7] Kim Y, Park J, Presley M. Selecting significant contextual factors and estimating
their effects on operator reliability in computer-based control rooms. Reliab Eng
Syst Saf 2021;213:107679.
[8] Liao H, Forester J, Dang VN, Bye A, Chang YHJ, Lois E. Assessment of HRA method
predictions against operating crew performance: part III: conclusions and
achievements. Reliab Eng Syst Saf 2019;191:106511.
[9] Podofillini L, Dang VN, Nusbaumer O, Dres D. A pilot study for errors of
commission for a boiling water reactor using the CESA method. Reliab Eng Syst Saf
2013;109:86–98.
[10] Yang J, Kim T, Kim J. Analysis of errors of commission for a CE type plant with the
advanced control room in the full power condition. Ann Nucl Energy 2017;105:
184–95.
[11] He X, Andersson C, Olsson A, Ljungbjörk J, Karlsson A, Nordlof L, Hellström P.
Errors of Commission in HRA–NPSAG Phase 1 project. In: Probabilistic Safety
Assessment and Management PSAM 14, September 2018; 2018.
[12] International Atomic Energy Agency. Attributes of Full Scope Level 1 Probabilistic
Safety Assessment (PSA) for Applications in Nuclear Power Plants. In: IAEA-
TECDOC-1804; 2016.
[13] U.S. Nuclear Regulatory Commission. Technical basis and implementation
guidelines for a technique for human event analysis (ATHEANA). 2000. NUREG-
1624, Rev. 1.
[14] Reer B. Review of advances in human reliability analysis of errors of
commission—Part 2: EOC quantification. Reliab Eng Syst Saf 2008;93(8):1105–22.
[15] Podofillini L, Mkrtchyan L, Dang VN. Quantification of Bayesian belief net
relationships for HRA from operational event analyses. In: Proceedings of PSAM
(Vol. 12); 2014.
[16] Kim JW, Jung W, Park J. A systematic approach to analysing errors of commission
from diagnosis failure in accident progression. Reliab Eng Syst Saf 2005;89(2):
137–50.
[17] Massaiu S, Fernandes A. Comparing operator reliability in analog vs. digital
human-system interfaces: an experimental study on identification tasks. In: PSAM
Topical Conference on Human Reliability, Quantitative Human Factors, and Risk
Management; 2017. p. 7–9.
[18] Kim Y, Choi SY, Park J, Kim JW. KAERI Technical report. 2021. KAERI/TR-8632/
2021.
[19] Agresti A. Foundations of linear and generalized linear models. New Jersey: John
Wiley & Sons; 2015.
[20] Kim Y, Kim J, Park J, Choi SY, Kim S, Jung W, Kim HE, Shin SK. KAERI technical
report. 2019. KAERI/TR-7607/2019.
[21] Swain AD, Guttmann HE. Nuclear Regulatory Commission. 1983. NUREG/CR-
1278.
[22] Chang JY, Cheng A. Case Study of Human Error Probability in Commercial
Aviation. In: International Topical Meeting on Probabilistic Safety Assessment and
Analysis (PSA 2015) Sun Valley, Idaho, USA April 2015; 2015.
[23] EPRI. Establishing minimum acceptable values for probabilities of human failure
events: practical guidance for probabilistic risk assessment. Palo Alto, CA: EPRI;
2010, 1021081. 2010.
[24] Whaley AM, Kelly DL, Boring RL, Galyean WJ. SPAR-H step-by-step guidance.
Idaho falls (ID): risk, reliability, and NRC programs department. Idaho National
Laboratory; 2011.
[25] Heinze G, Wallisch C, Dunkler D. Variable selection–a review and
recommendations for the practicing statistician. Biom. J. 2018;60(3):431–49.
[26] Čepin M. DEPEND-HRA—A method for consideration of dependency in human
reliability analysis. Reliab Eng Syst Saf 2008;93(10):1452–60.
Y. Kim et al. Reliability Engineering and System Safety 227 (2022) 108727

[27] Paglioni VP, Groth KM. Dependency definitions for quantitative human reliability [29] Kichlinea M, Xing J, Chang YJ. Dependency analysis using the integrated human
analysis. Reliab Eng Syst Saf 2022;220:108274. event analysis system human reliability analysis methodology. In: Probabilistic
[28] Arigi AM, Park G, Kim J. Dependency analysis method for human failure events in Safety Assessment and Management PSAM 16, Honolulu, Hawaii; 2022.
multi-unit probabilistic safety assessments. Reliab Eng Syst Saf 2020;203:107112.