Safety Science 109 (2018) 130–143

Contents lists available at ScienceDirect

Safety Science
journal homepage: www.elsevier.com/locate/safety

System safety assessment based on STPA and model checking T

⁎
Alheri Longji Dakwat , Emilia Villani
Instituto Tecnológico de Aeronáutica (ITA), Pça. Mal. Eduardo Gomes, 50, São José dos Campos CEP 12228-900, Brazil

A R T I C LE I N FO A B S T R A C T

Keywords: Due to the current pace of technological growth, the management of system safety has evolved with complex
Model checking causes of accidents that are often beyond the identification of traditional safety assessment techniques. Recently,
STPA the hazard analysis tool Systems Theory Process Analysis (STPA) has emerged as an approach to improve safety
Safety assessment of modern complex systems in concert with other hazard analysis tools. However, the effectiveness of STPA is a
UPPAAL
debatable issue in the industry and efforts towards incorporating some level of formalization in STPA steps are
welcome. In this direction, this work presents a method for combining STPA and model checking, in order to
provide a formal and unambiguous representation of the system under analysis and the threats identified by
STPA. A practical case study of a robotic flight simulator is presented as an example of the proposed method. The
results achieved with the proposed approach indicates that the merging of the two techniques improves the
knowledge about the system under design and the consistence of the design changes proposed to tackle the safety
constraints identified in STPA.

1. Introduction
The computer revolution has made significant impact in the society,
especially through embedded systems. Embedded systems have made
possible the incorporation of many new functionalities that are im-
plemented in software. As we increase the complexity of the systems
under design, traditional bottom-up or top-down safety assessment
techniques, such as Failure Mode and Effect Analysis (FMEA) and Fault
Tree Analysis (FTA), become insufficient to assure the product safeness
(Stringfellow et al., 2010). One limitation is the difficult of exploring all
the possible scenarios that may arise from the combination of the
components’ behavior when interacting with software products and
human beings. In this context, the hazard analysis tool Systems Theory
Process Analysis (STPA) has emerged as an approach for improving
safety of modern complex systems in concert with other hazard analysis
tools. The traditional approaches to safety, such as HAZOP, FMEA, and
FTA, were developed for systems that were built more than 50 years
ago. Their focuses is to address failures based on chain of events related
to component malfunction. Modern embedded systems have developed
new paths to accidents based on complex errors that emerge even when
everything is working according to its specification (Leveson, 2011). A
systematic review on STPA observed that there has been published
works on STPA since 2000s and a sharp increase from 2011 in areas
that include aviation, automotive, defense, software, space, medical
and environment (Lahoz, 2016). This survey underscores the gaining
relevance and acceptance of STPA as a safety tool.
Despite being a promising approach, STPA has not yet been ex-
tensively adopted by industry or recommended by certification autho-
rities. Among the limitations of STPA, are the lack of formalism
(Harkleroad et al., 2013) and the potential dependence of the results on
the experience of the person that applies the method. In order to
overcome these limitations, this work proposes the use of STPA com-
bined with model checking, a formal verification technique. The chal-
lenges emerging from the exponential growth of embedded systems is
resulting in an increasing interest in formal methods. Industry is cur-
rently considering the adoption of formal techniques, such as model
checking and theorem proving, to complement the more common
verification approaches of simulation and testing. These techniques are
gaining grounds through industrial case studies, reinforcing the benefits
of using formal methods (Clarke and Wing, 1996; Benaben et al., 2002).
In the current work, automata are adopted as a formal language to
model the behavior of the system components, including software,
while model checking is used to check the consistence of the safety
constrains that are obtained from the STPA approach. The contribution
of this work is safety assessment approach that combines STPA and
model checking. From STPA, the proposed approach inherits the sim-
plicity and ability of identifying scenarios that can lead to hazard si-
tuations. Complementing STPA, model checking contributes with the
complete exploration of the system state space. It complements the
identification of unsafe control actions and verifies the effect of the
safety constraints in inhibiting hazard situations.
A practical case study of the embedded system of a flight simulator

⁎
Corresponding author.
E-mail addresses: alheri.dakwat@airforce.mil.ng (A.L. Dakwat), evillani@ita.br (E. Villani).

https://doi.org/10.1016/j.ssci.2018.05.009
Received 7 June 2017; Accepted 15 May 2018
Available online 05 June 2018
0925-7535/ © 2018 Elsevier Ltd. All rights reserved.
A.L. Dakwat, E. Villani
prototype is used to evaluate this proposal. The simulator, known as
SIVOR, explores the use of an industrial robotic manipulator to provide
the acceleration feelings to the pilot. Due to the innovative nature of the
SIVOR project, it offers a rich field of analysis of new types of hazards.
This paper is organized as following. Section 2 discusses approaches
to safety based on the concepts of STPA and Model Checking using
UPPAAL. Section 3 presents the workflow of the proposed method and
Section 4 details its application to the SIVOR case study. Finally, Sec-
tion 5 draws some conclusions.
2. Literature review
A systematic literature review on STPA (Lahoz, 2016) indicated a
positive trend in its acceptance. The review presented a summary of
STPA applications since the early 2000s. These included 23 methods, 37
approaches, 8 tools and about 176 case study applications, out of which
the highest numbers were in aviation, medicine, automotive and space
industries and rising. From this extensive review, we selected 3 papers
that went further to consider some form of scientific verification
through formal methods. We then complement this set with 15 works
on system safety based on STAMP/STPA published from 2010 in the
Science Direct and IEEE databases after 2010, resulting in a final set of
18 analyzed works (Abdulkhaleq et al., 2015; Lopes and Hirata, 2015;
Chiesi, 2016; Asplund et al., 2012; Cody et al., 2011; Takuto et al.,
2011; Suo et al., 2011; Haruka et al., 2011; Sulaman et al., 2014; Gerrit
et al., 2016; Asim et al., 2017; Kleve, 2016; Maria et al., 2016; Robert
et al., 2017; Petit, 2004; Rejzek et al., 2015; Sun and Zhong, 2013;
Shuai and Deming, 2014; Asim and Stefan, 2015). Among these works,
we discuss in this section those that were more closely related to our
proposal.
The work of Asplund et al. (2012) presented an analysis of the
implementation of STPA on an industrial tool chain. Some major dif-
ficulties observed included how to identify basic risks for new contexts,
defining control structures and limiting the domains. The authors
identified the need of tools to support STPA as well as perform simu-
lations through which STPA users can explore the consequences of in-
adequate control in a control structure and equally facilitate the ap-
propriate strategies for updating the system.
Another similar effort, by Lopes et al. (Baier and Katoen, 2007),
proposed a rule-based approach to perform Step 1 of STPA. The pro-
posal was validated based on its application to the single controller of
an automated door system of a train. Although the authors conclude
that the result was encouraging, the application of the rule-based ap-
proach on larger systems with multiple controllers and Step 2 was
uncertain.
In another work, Abdulkhaleq et al. (Leveson, 2013) proposes the
use of a software tool (ASTAMP) to perform STAMP (CAST and STPA)
activities. An extended version of the software, XSTAMP has continued
to evolve with the extension to perform verification. The possibility of
combining UML and STPA (Kleve, 2016) also presents a practical ap-
proach to safety based on the integration of system and safety en-
gineering.
Two other published works (Maria et al., 2016) and (Petit, 2004)
combine formal methods with STPA. Sun & Zhong (Maria et al., 2016)
considered using STPA thinking combined with a technique to convert
natural language to automaton using a traffic light as example. Asim &
Stefan (Petit, 2004) proposes to integrate STPA with state machine
analysis. In this approach, they used STPA to determine the control
actions of a model and then constructed a finite state machine of the
controller with state variables to examine each potential hazardous
control action and the effect of combined system states. One of the
challenges identified in this work as the need of developing an appro-
priate model of the real system.
Based on the literature review, we believe that the challenges of
STPA are mainly due to the lack of a scientific method to address the
hypothetical nature involved in its application. This is further
131
Safety Science 109 (2018) 130–143
Fig. 1. Assumptions of proposed method.
confounded by the problem of how and what is to be verified in STPA
when using UPPAAL. To resolve this, we considered that both STPA and
UPPAAL have limitations when applied independently. The hypothe-
tical quality of STPA allows it to accommodate unforeseen hazards
while model checking is restricted to the system model, not the actual
system (Rejzek et al., 2015). That is, STPA looks at hypothetical actions
(WHAT IF?) as a means to eliminating hindsight bias (Leveson, 2011)
while model checking looks at available information (WHAT IS?). The
outcome of this union is considered as the answer, (WHAT WILL BE) as
illustrated in Fig. 1. These assumptions could also be viewed from the
perspectives of human and machine limitations. That is, the human can
make predictions that may be right or wrong but the computer cannot.
Likewise, the computer cannot predict the future as it can only make
calculated analysis based on existing data. In view of these challenges,
the problem considered is how to achieve a scientific method that can
facilitate the combination STPA and model checking or validating the
results of STPA using scientific methods.
3. The proposed method
STPA is one of the hazard analysis tools of STAMP conceived by
Nancy Leveson in the early 2000s. STPA is performed in 2 steps
(Leveson, 2011) and its goal is to identify scenarios leading to hazards
and then define safety constrains that the system under design must
fulfill. STPA identifies a larger set of causal factors than other safety
techniques, many of them not involving failures or unreliability of
components (Sun and Zhong, 2013). It also provides appropriate in-
formation to guide the design process, instead of requiring a design to
exist before the analysis can start (Leveson, 2011).
On the other hand, model checking is a verification technique that
explores all possible system states in an exhaustive manner (Rejzek
et al., 2015). To confirm that a given system model truly satisfies a
certain property, model checking must examine all possible system
scenarios in a systematic manner (Shuai and Deming, 2014). The 3
basic steps in model checking involves building the model, formalizing
the properties to be verified and using the model checker tool to check
if the specified properties are true or false in the dynamic behavior of
the model. In this work, we use the model checker UPPAAL, which
requires the system modelling as a network of timed automata (Asim
and Stefan, 2015).
An appropriate workflow to combine both methods is based on the
premise that both complement each other based on their character-
istics. For example, STPA requires a preliminary assessment that defines
elements and control actions of the system. These elements could be
assumed to correspond to templates and channels in UPPAAL. The
workflow of the proposed method is organized in 6 steps that are il-
lustrated in Fig. 2 and discussed in subsequent paragraph.
The inputs and outputs of each step are illustrated in Fig. 3. The
workflow starts with Step 1, from STPA. The output of this step yields
the system goals, elements (human, software and hardware), list of
hazards and accidents, list of control actions, the control structure and a
description of system operation (UML charts may come in handy during
this process depending on the complexity of the system). Based on these
outputs, Steps 2 and 3 can be performed concurrently. Step 2, which is
A.L. Dakwat, E. Villani
Fig. 2. Workflow of proposed method.
also dependent on Step 1, involves the modelling and simulation of the
system normal operation. In UPPAAL, the templates and channels
correspond to elements and controlled actions from STPA. Step 3 also
involves the process of determining the unsafe controlled actions that
are derived from the list of controlled actions in Step 1, following the
STPA approach. Then, Steps 4 and 5 must be performed in parallel in
order to result in a design proposal that fulfills the safety constraints of
STPA. The reviewed controlled structure of Step 4 is based on inputs
from unsafe controlled actions from Step 2, appropriate safety con-
straints, as well as feedback from Step 5. This step continue to evolve
along with Step 5. For Step 5, the UPPAAL model of the system during
normal operation is updated with the system hazards and the associate
design modifications to couple with the hazards. Based on the system
requirements, these inputs are reflected in the simulated model through
guards, channel prioritization, updates, clocks and synchronized ac-
tions. Both Steps 4 and 5 depend on each other’s output. Step 6 involves
the verification of safe constraints on the system design. In UPPAAL this
is achieved through the verification of safety, liveness and reachability
properties.
4. Case study
4.1. The SIVOR prototype simulator
The SIVOR (SImulador de VOo de base Robótica, robotic flight si-
mulator) is a project under development at ITA in partnership with
EMBRAER, and with financial support from the Sao Paolo Research
Foundation FAPESP. It aims at building a flight simulator using as
motion system a 6-degree of freedom anthropomorphic robot, instead
132
Safety Science 109 (2018) 130–143
of the traditional Stewart platform based on electrical or hydraulic
actuators. The motion system should provide the pilot with sensations
of acceleration that are similar to those felt in a real flight. The SIVOR
simulator is illustrated in Fig. 4.
The motion platform is a KUKA robot with a payload of 500 kg. The
pilot seat is equipped with inceptors (throttle, sidestick and rudder
pedals) and the vision system is a TV. Additionally, a black out cov-
erture isolates the pilot from the external environment. The X-Plane
software tool is responsible for generating the image for the visual
system and for processing the aircraft aerodynamics. The wash-out
filter, an in-house developed software package, is responsible for con-
verting the aircraft accelerations into angular and linear increments to
be sent to the robot. Communication between the washout filter and the
robot controller is performed using RSI interface, provided by the robot
manufacturer.
SIVOR is a safety critical application because robots are not de-
signed to operate in contact with humans or as a flight simulator.
Industrial robots usually operate in an isolated environment, such as a
cage or a zone monitored by presence detectors. Moreover, they are
usually programmed to perform the same sequence of commands,
which are extensively verified, both by simulation and step-step ex-
ecution, prior to operation. In the case of the flight simulator, the tra-
jectory of the robot is unknown and depends on the interaction with the
pilot. Moreover, for obvious reason, the pilot cannot be isolated from
the robot. An accident could put at risk the life of the pilot and result in
significant equipment loss.
A.L. Dakwat, E. Villani Safety Science 109 (2018) 130–143

Fig. 3. Inputs and outputs of the steps.

Fig. 4. SIVOR flight simulator.

4.2. Step 1- system analysis based on STPA
In step 1, the preliminary analysis based on STPA highlights the
goal of the system, system elements, the control actions by system
elements, likely hazards and accidents as well as the sequence of op-
erations.
The goal of the SIVOR prototype simulator is to act as a testing
platform for the development of control laws, which ensure the robot
motion so that the pilot realizes the robot's movements as being re-
presentative of aircraft movement in the various stages of flight and in
133
perfect harmony with the commands that were applied.
The accidents considered in the safety analysis include death or
injury to human elements [A1] and physical damage to equipment
[A2]. The potential hazards that could lead to these accidents are the
following:
• [H1] The robot goes out of its safe workspace;
• [H2] The pilot is subjected to extreme acceleration;
• [H3] Equipment subjected to extreme acceleration;
• [H4] Encroachment of people or equipment into robot’s workspace.
A.L. Dakwat, E. Villani
Fig. 5. Phases of SIVOR operation.
In order to understand the system behavior, the SIVOR operation
was organized in 5 phases as illustrated in Fig. 5.
The primary actors or elements of the system are:
• Pilot, the person that sits on the cockpit of the flight simulator and
commands it;
•
Robot, which moves the simulator cockpit and the pilot in response
to commands from the robot controller;
• Robot controller, a computer based system that receives position
increments from the SIVOR controller and sends it to the robot;
• KCP (Kuka Control Panel) operator, an engineer that interacts with
the Robot controller through a remote panel;
• X-Plane, a software tool that generates the image that is displayed in
the cockpit and calculates the aircraft aerodynamics based on the
pilot input;
•
SIVOR controller, a computer based system that applies the wash-
out filter, and provides position increments to the robot controller
based on the aircraft aerodynamics.
• SIVOR operator, an engineer that operates the SIVOR controller.
These elements perform various concurrent actions that define the
sequence of operations of the SIVOR from power-up to shut down.
Phases 1 and 2 are carried out mostly by the KCP operator and they
involve powering on the flight simulator without the pilot in the cockpit
and subsequent cockpit preparation for the simulation with the pilot
onboard. Phase 3 corresponds to the setup of the communication be-
tween SIVOR controller and the robot controller. Phase 4 is the flight
phase. Lastly, Phase 5 involves the finalization of simulation.
A total of 33 control actions were identified for the normal opera-
tion of the SIVOR flight simulator. These actions are illustrated in the
control structure of Fig. 6 and described in the next section.
4.3. Step 2 – UPPAAL modelling (normal operations)
Step 2 involved the modelling of the SIVOR system in UPPAAL on
normal operation mode. To achieve this, the identified control actions
and elements were mapped into UPPAAL as channels and templates
Table 1). The control structure is used to guide and trace syncronisation
of channels. The system model is composed of 7 automata (KCP op-
erator, robot controller, SIVOR operator, SIVOR controller, X-Plane,
robot and pilot). The modelling activity is considered successful when
all control actions are included in the model, all states of the automata
are reachable, the synchronization of the automata is consistent, and
the model presents no deadlock (see Table 2).
The automata represent the dynamics of each element of the system
134
Safety Science 109 (2018) 130–143
and how they interact through common events (UPPAAL channels).
The automaton of the KCP operator is presented in Fig. 7. Automaton
of the KCP operator.
. The urgent states (indicated with an ‘u’) are the related to inter-
mediary states between two consecutive actions that should be per-
formed without a significant time interval between them.
The SIVOR operator manages and controls the SIVOR controller and
X-Plane software. He receives instructions from the pilot to start the X-
Plane software. His duties of ‘starting the SIVOR controller’ up to ‘en-
abling robot movement through the RSI’ as well as from ‘stopping the
RSI’ to stopping X-Plane’ as urgent states. Fig. 8 shows all procedures of
the SIVOR operator. The SIVOR controller Fig. 9) provides the angular
and linear increment data for the robot controller based on input from
the X-Plane.
The pilot automaton Fig. 10) has also a preparation sequence of
events and then, during flying phase, it is responsible to generate the
controlling input through the sidestick.
The X-Plane template Fig. 11) provides visual feedback of aircraft
orientation to the pilot. It has only 4 states: off, on, ready and proces-
sing. The X-Plane is switched on by the SIVOR operator. The pilot then
performs a communication test to confirm the system is ready for si-
mulation. From this state, it receives the inceptor data from the pilot,
generates the aircraft data (roll, pitch, yaw) and sends it to the SIVOR
controller. It can return to the initial state from both the ‘processing’
and ‘ready’ states.
The robot controller Fig. 12) receives the angular and linear incre-
ment data from the SIVOR controller to drive the robot motors. This is
made possible through the RSI communication protocol. The robot
controller also runs the KRL which is a preset KUKA robot language
program that the KCP operator uses to move the robot based on default
preset sequence. Such preset movements include moving robot to home
or start positions. The robot controller also receives the position data of
the robot which is used to provide feedback alert when the robot
reaches the desired position.
The robot template Fig. 13) defines the behavior of the robot based
on the input provided by the robot controller. The basic sequence of the
robot after being forced out of its initial state is goes to the ‘home po-
sition’ then ‘start position’ to enable preflight and flight activities and
lastly returning or ‘going home’.
During the system modelling, several errors and deadlocks were
generated but eventually cleared. Among the features of UPPAAL that
can be used in the debugging activity, we highlight the use of the trace
feature and sequence diagram, as well as the formal verification of
properties related to the consistence among automata.
4.4. Step 3 – Unsafe control actions
In Step 3, according to STPA, the control actions are used to identify
the unsafe control actions based on the four scenarios: when it is not
provided, when it is provided, when it is provided too soon and when it
is provided too late/too long. As an example, we present the unsafe
control action analysis for Phase 2.
The unsafe control actions identified in all the 5 phases were sum-
marized in the following five itens:
UCA 1. Robot moves when there is an obstruction in robot work-
space;
UCA 2. KCP operator choose a wrong KRL program;
UCA 3. Robot may move while pilot is entering or without seatbelts,
either because events of the simulation procedure happens before it
should, or because the pilot is late;
UCA 4. Abrupt stop of robot with large de-acceleration;
UCA 5. Providing wrong values of command to the robot.
Following the Step 2 of STPA, we analyze each unsafe control ac-
tion, propose safety constraints and design changes for each of them.
A.L. Dakwat, E. Villani Safety Science 109 (2018) 130–143

Fig. 6. Control structure of SIVOR flight simulator under normal operation.

UCA 1. Robot moves when there is an obstruction in robot work- workspace is empty when it is not.
space The corresponding safety constraint is:
The UCA 1 results from an incorrect process model of the robot
controller, which allows the robot to move assuming that the robot SC 1. The robot controller shall be able to detect and stop the robot

Table 1
Control actions based on STPA and corresponding UPPAAL channels.
Control actions UPPAAL channel

1. KCP operator turns on robot controller Turn_ON

2. KCP operator runs the KRL program on the robot controller Run_KRL
3. Robot controller moves robot to home position Move_home, Robot_home
4. Robot controller informs KCP operator that the robot has reached home position Home_ack
5. KCP operator stops KRL program on the robot controller Stop_KRL
6. KCP operator requests pilot to enter the cockpit Req_pilot
7. Pilot enters the cockpit, fastens the seat belt and puts the headphone Pilot_OK
8. Pilot make verbal communication with the SIVOR operator Call_SIVOR_optr
9. SIVOR operator starts X-Plane Start_XPlane
10. Communication test among pilot, SIVOR operator, and X-Plane Comm_XPlane, Comm_SIVOR_optr
11. SIVOR operator starts SIVOR controller Start_SIVOR_ctrl
12. SIVOR operator informs KCP operator to start KRL program Call_KCP_optr
13. KCP operator runs KRL program on robot controller Run_KRL
14. Robot controller moves robot to start position Move_start_pos, Robot_start_pos
15. Robot controller starts RSI and checks communications with SIVOR controller Check_RSI
16. SIVOR operator checks communications in SIVOR controller and starts of simulation Enable_robot
17. SIVOR operator informs pilot that simulation has started Start_sim
18. Pilot maneuvers the aircraft (inceptors sends commands to X-Plane) Inceptor_data
19. X-Plane sends aircraft motion variables to SIVOR controller Aircraft_data
20. SIVOR controller provides the angular/linear increments to the robot controller Robor_incr
21. Robot controller drives the robot motors Move_robot
22. Robot sends the current position to the robot controller Robot_pos
23. Pilot requires to end simulation Stop_sim
24. SIVOR operator stop the communication between the SIVOR controller and the robot controller; Stop_RSI_comm
25. SIVOR operator inform KCP operator Info_KCP_optr
26. KCP operator executes KRL program on robot controller to bring robot to the home position. Return_home
27. Robot controller moves the robot to the home position and informs KCP operator Move_home, Robot_home
28. Robot controller informs KCP operator that the robot has reached home position Home_ack
29. The SIVOR operator shuts down the SIVOR controller Stop_SIVOR_ctrl
30. The SIVOR operator shuts down X-Plane Stop_XPlane
31. The KCP operator informs the pilot to leave the cockpit Pilot_out
32. The pilot leaves the cockpit and informs KCP operator No_pilot
33. The KCP operator turns off the robot controller Turn_OFF

135
A.L. Dakwat, E. Villani Safety Science 109 (2018) 130–143

Table 2
Identification of unsafe control actions for Phase 2 – Pilot preparation.
Control actions Not providing causes hazard Providing causes Provided too soon Stopped too late
hazard

(a) (b) (c) (d) (e)

1. KCP operator request pilot to enter the cockpit; NA NA Robot may move while pilot is NA
entering [H2]
2. Pilot enters cockpit, fastens the seat belt and Unfasttened seat belt [H2] NA Robot may move while pilot is NA
puts the headphone; entering [H2]
3. Pilot make verbal communication with the Start simulator while pilot is entering or NA Start simulator while pilot is entering NA
SIVOR operator. before fastting seat belt [H2] or before fastting seat belt [H2]
4. SIVOR operator starts X-Plane NA NA NA NA
5. Communication test between pilot and SIVOR NA NA NA NA
operator, using X-Plane

Fig. 7. Automaton of the KCP operator.

Fig. 8. Automaton of the SIVOR operator.

in the case an intruder invades the robot workspace. going out of its workspace.
The corresponding safety constrain are:
UCA 2. KCP operator choose a wrong KRL program
The UCA 2 results from an incorrect process model of the KCP op- SC 2. The robot controller shall not move the robot out of its
erator, which either selects the wrong KRL program or select a KRL workspace;
program that is incorrect, resulting in high acceleration or the robot SC 3. The robot controller shall not impose high acceleration on the

136
A.L. Dakwat, E. Villani Safety Science 109 (2018) 130–143

Fig. 9. Automaton of the SIVOR controller.

Fig. 10. Automaton of the pilot.

operator or the SIVOR operator, which believes the pilot is ready (or
out of the cockpit), when it is not.
The corresponding safety constrain are:

SC 4. The robot shall not move when the pilot is on-board but
without seatbelt;

UCA 4. Abrupt stop of robot with strong de-acceleration

As stopping the robot is the safe action in many dangerous situa-
tions, the abrupt interruption of the robot movement cannot be ex-
cluded from the SIVOR solution. In order to avoid this unsafe control
action, the following safety action is specified:

SC 5. The speed of the robot should be limited in order to avoid

Fig. 11. Automaton of the X-Plane.
strong de-acceleration.

robot when the pilot is on-board; UCA 5. Providing wrong values of command to the robot
The safety constraint for UCA 5 is same as UCA 2.
UCA 3. Robot may move while pilot is entering or without seat-
belts, either because events of the simulation procedure happens be-
fore it should, or because the pilot is late;
The UCA 3 results from an incorrect process model of the KCP

137
A.L. Dakwat, E. Villani
Fig. 12. Automaton of the robot controller.
4.5. Step 4 – Reviewed control structure
The safety constrains proposed in Step 3 resulted in the incorpora-
tion of a set of design modifications, as indicated in Table 3.
The design changes C and E are internal filters of the robot con-
troller and the SIVOR controller. They do not affect the control struc-
ture of the SIVOR project. The other design changes (A, B, D and F)
requires the definition of what should be done after the robot movement
is interrupted. For this purpose, an additional operation phase is in-
troduced. Phase 6 is characterized by the occurrence of an emergency
stop, initiated by the pilot, the KCP operator, the workspace sensor or
the seatbelt sensor.
The emergency is activated by one of the following events:
(1) Pilot activates the emergency button while robot is moving;
(2) KCP operator release the deadman button of the KCP smartpad
while robot is moving;
(3) Seatbelt sensor is activated while robot is moving;
(4) Workspace sensor is activated while robot is moving;
(5) The emergency treatment includes the following new control ac-
tions:
Fig. 13. Automaton of the robot.
138
Safety Science 109 (2018) 130–143
(6) Robot controller cut the power, activates the brakes and send an
error alert to the KCP operator;
(7) Robot controller keeps the communication with SIVOR controller;
(8) KCP operator check the situation and decides if simulation should
be finalized or continued.
(9) Pilot release the emergency button;
(10) KCP operator press the deadman button of the KCP smartpad;
(11) Seatbelt sensor becomes active;
(12) Workspace sensor becomes inactive;
(13) KCP operator clear the screen and decides if simulation should be
finalized or continued;
(14) The KCP operator restarts the simulation;
If the KCP operator decides to continue simulation it returns to
Phase 4. If he decides to interrupt simulation, it goes to Phase 5.
The updated control structure of the SIVOR flight simulator, now
including also the prevention of unsafe control actions is presented in
Fig. 14.
The new control actions should also be analyzed as potential unsafe
control actions and eventually new design changes may be introduced
to couple with new unsafe situations, in a cyclical process performed
A.L. Dakwat, E. Villani Safety Science 109 (2018) 130–143

Table 3
Safety constraints and corresponding design changes.
Safety constraints Design changes

SC 1. The robot controller shall be able to detect and stop the robot in the
case an intruder invades the robot workspace.
SC 2. The robot controller shall not move the robot out of its workspace;
SC 3. The robot controller shall not impose high acceleration on the robot
when the pilot is on-board.
SC 4. The robot shall not move when the pilot is on-board but without
seatbelt.
SC 5. The speed of the robot should be limited in order to avoid strong de-
acceleration.
A. Emergency button available for the KCP operator to interrupt the robot movement.
B. Installation of sensor in the workspace that detect the entrance of an external person and
interrupts the robot movement.
C. Configuration of joint limits in the robot controller.
A. Emergency button available to the KCP operator to interrupt the robot movement.
D. Emergency button available for the pilot to interrupt the robot movement.E. Limitation of the
maximum increment of the robot position at each step of the SIVOR controller.
F. Installation of seatbelt sensor.
E. Limitation of the maximum increment of the robot position at each step of the SIVOR controller.

together with Step 5, until a safe design is reached.

4.6. Step 5 – SIVOR UPPAAL model with emergencies

In Step 5, the UPPAAL model is modified to include the design

changes proposed in Step 4 to couple with unsafe control actions
identified in Step 3. Step 5 also serves as a continuous feedback to Step
3. In the case of the SIVOR flight simulator, the automata must include
the generation of emergencies (events 34–38) and their treatment
(events 39–46). The following automata are modified: robot, emer-
gency generator, KCP operator, the pilot and robot controller.
An additional automata is also introduced: the emergency_generator
Fig. 15. Automaton of the emergency generator.
Fig. 15. It includes the unsafe actions (events 34–38) that trigger the
emergency procedures. We observe that the four events are associated
to a single channel (start_emergency), as they have a similar effect on
the system, i.e. they interrupts other activities and halt the robot

Fig. 14. SIVOR control structure, including design changes to comply with safety constraints.

139
A.L. Dakwat, E. Villani Safety Science 109 (2018) 130–143

Fig. 16. Updated automaton of the robot, with emergency events.

Fig. 17. Updated automaton of the KCP operator, with emergency events.

140
A.L. Dakwat, E. Villani Safety Science 109 (2018) 130–143

Fig. 18. Updated automaton of the robot controller, with emergency events.

Fig. 19. Updated automaton of the pilot, with emergency events.

movement. Similarly, events 39–42 are also associated to a single prescribed in Phase 6. They corresponds to the following cases: when
channel (stop_emergency). the robot is moving to home position, when the robot is moving to start
The updated model of the robot is presented in Fig. 16. Four sce- position, during the flying simulation and when the robot is returning
narios are considered unsafe and requires the emergency procedures to home position.

141
A.L. Dakwat, E. Villani Safety Science 109 (2018) 130–143

Table 4
Verification of the model correctness.
CTL Property Description

A[] not deadlock All states are reachable

A < > KCP_operator.Robot_home_1
KCP_operator.Robot_home_1 – > KCP_operator.OFF
A[] (KCP_operator.OFF and time > 0) imply (SIVOR_operator.Standby)
A[] Robot.Moving_incr imply Robot_controller.going_home or Robot_controller. Going_start_pos or
Robot_controller.DriveRobotMotors or Robot_controller.Returning _home
Emergency_generator.Treating – > (Robot.Halted_1 or Robot.Halted_2 or Robot.Halted_3 or
Robot.Halted_4)
A[] Pilot.Waiting_sim imply (!Robot.Going_home_ini or !Robot.Moving_incr or
!Robot.Going_start_pos or !Robot.Going_home_end)
A[] Pilot.Waiting_to_leave imply (!Robot.Going_start_pos or Robot.Going_home_end)
Actions 2, 3 and 4 will be performed by the KCP operator
After performing it, the KCP operator will return to initial state
When KCP operator is off, SIVOR operator is on standby for next transition
when t > 0
Robot moving imply the robot controller is initiating motion for going to
home position, start position, driving robot motors or returning home
Robot has been halted in any of the 4 prescribed halt states if the
emergency generator has initiated a treatment procedure.
When pilot is waiting in the cockpit to start simulation, the robot is not
going to home position, going to start position, accepting data increment,
or going to home position
Pilot waiting to leave cockpit imply robot is in home position

Table 5
Verification of safety constraints.
Safety Constraints Design Changes CTL Property

SC 1. The robot controller shall be able to
detect and stop the robot in the case an
intruder invades the robot workspace
SC 2. The robot controller shall not move the
robot out of its workspace
SC 3. The robot controller shall not impose
high acceleration on the robot when the
pilot is on-board
SC 4. The robot shall not move when the pilot
is on-board but without seatbelt
SC 5. The speed of the robot should be limited
in order to avoid strong de-acceleration
A. Emergency button available for the KCP operator Emergency_generator.Treating – > (Robot.Halted_1 or Robot.Halted_2 or
to interrupt the robot movement Robot.Halted_3 or Robot.Halted_4)
B. Installation of sensor in the workspace that
detect the entrance of an external person and
interrupts the robot movement
C. Configuration of joint limits in the robot (Internal, not represented in the UPPAAL model)
controller
A. Emergency button available to the KCP operator Emergency_generator.Treating – > (Robot.Halted_1 or Robot.Halted_2 or
to interrupt the robot movement Robot.Halted_3 or Robot.Halted_4)
D. Emergency button available for the pilot to
interrupt the robot movement
E. Limitation of the maximum increment of the
robot position at each step of the SIVOR controller
F. Installation of seatbelt sensor Emergency_generator.Treating – > (Robot.Halted_1 or Robot.Halted_2 or
Robot.Halted_3 or Robot.Halted_4)
E. Limitation of the maximum increment of the Robot_controller.Resuming_sim – > (Robot_controller.Request_home and
robot position at each step of the SIVOR controller, Robot.Halted_3) or (Robot_controller.Position_updated and
based on the measurement of the robot position Robot.Halted_3)

The updated model of the KCP controller is presented in Fig. 17. The
KCP operator also has to acknowledge the emergency and, in the case of
the flight phase, decides between resume the simulation or interrupt it.
The updated model of the robot controller is presented in Fig. 18.
The robot controller acts on the error alert generated by the source of
the emergencies. During each situation, the robot controller processes
the decision from the KCP operator to interrupt or resume the simula-
tion.
Finally, the updated model of the pilot is presented in Fig. 19. When
the KCP operator decides to finish the simulation, it inform the pilot,
which, in the appropriate moment, has to leave the cockpit.
4.7. Step 6 – Safe system specification
system design.
Concluding the case study, we consider that the proposed method of
combining STPA and model checking using UPPAAL has been suc-
cessfully applied in the SIVOR flight simulator. The entire process re-
quired some effort to make the STPA analysis consistent with the
UPPAAL model, but they both contributed to each other. The need of
formalizing the system behavior in an automata model arose a number
of doubts, which should then be clarified in order to proceed with the
application of the proposed approach. As a result, the knowledge about
the system and possible dangerous scenarios was substantially in-
creased. On the other hand, the STPA steps resulted in identification of
dangerous situations that were not considered when initially modelling
the system.

The last step of the proposed approach consists of formally verifying
that the system model with emergency events is correct and the safety
constraints are satisfied, i.e., the unsafe control actions do not violate
the safety constraints considering the proposed design changes. Both
liveness, reachability and safety properties are used for this purpose.
Reachability properties checks that all the model states are even-
tually reached, or are reachable. Some examples of reachability prop-
erties used to verify the model correctness are presented in Table 4.
The safety constraints are the restrictions placed on unsafe control
actions by controllers of the SIVOR system. Any violations will make
the system unsafe. In order to assure that the system behaviour is in line
with design specifications, the design changes introduced in the system
are verified by CTL properties, as presented in Table 5. Any observed
inadequacies would imply the need for appropriate revision of the
5. Conclusion
The lack of formalization of STPA has become a hindrance to its
acceptance as an industry standard for hazard analysis. In order to
contribute to this issue, this work proposes a method that combines
STPA with model checking. The method comprises 6 steps and was
applied a robotic flight simulator.
The computational verification was achieved by modeling the entire
flight simulator in UPPAAL. This was possible because requirements of
STPA Step 1 such as the system elements and control actions were easily
mapped as templates and channels in UPPAAL. This further made it
possible to simulate the system operation and perform verification to
confirm that the system behavior was correct. Having achieved a model
of the system under normal operation, the control actions were further

142
A.L. Dakwat, E. Villani
used to identify unsafe controlled actions as defined in STPA. The
identified unsafe control actions resulted in the revision of the flight
simulator design. The design changes were then used to update the
UPPAAL model. Finally, the safety constraints were verified in the re-
vised model.
The results achieved with the proposed approach indicates that the
merging of the two techniques improves the knowledge about the
system under design and the consistence of the design changes pro-
posed to tackle the safety constraints identified in STPA.
The method presented in this work underscores the importance of
safety and system engineering working in tandem during product de-
velopment.
Acknowledgement
The authors acknowledge the financial support of FAPESP and
CNPq.
References
Abdulkhaleq, A., Wagner, S., Leveson, N., 2015. A comprehensive safety engineering
approach for software- intensive systems based on STPA. In: 3rd European STAMP
Workshop, Amsterdam.
Asim, A., Stefan, W., 2015. Integrating State Machine Analysis with System-Theoretic
Process Analysis. [Online]. Available: < https://www.researchgate.net/publication/
265508076_Integrating_State_Machine_Analysis_with_System-Theoretic_Process_
Analysis>.
Asim, A., Daniel, L., Stefan, W., Jürgen,R., Norbert, B., Ludwig, R., Thomas, R., Hagen, B.,
2017, A systematic approach based on stpa for developing a dependable architecture
for fully automated driving vehicles. In: 4th European STAMP Workshop.
Asplund, F., El-Khoury, J., Torngren, M., 2012. Safety-Guided Design through System-
Theoretic Process Analysis, Benefits and Difficulties. In: 30th International System
Safety Conference, Atlanta, USA.
Baier, C., Katoen, J.-P., 2007. Principles of Model Checking. MIT Press, Cambridge.
Benaben, F., Larnal, M., Pignon, J.P., Magnier, J., 2002. A process for improving multi-
technology system high level design: modeling, verification and validation of com-
plex optronic systems. In: IEEE International Conference on Systems, Man, and
Cybernetics.
Chiesi, S.S., 2016. STPA application for safety assessment of generic missile systems. In:
Reliability and Maintainability Symposium (RAMS).
Clarke, E.M., Wing, J.M., December 1996. Formal methods: state of the art and future
directions. ACM Comput. Surveys 28 (4), 627.
Cody, F., Takuto, I., Yuko, M., Haruka, N., Masa, K., Nobuyuki,H., John, T., Leveson, N.,
2011. Safety Guided Design of Crew Return Vehicle in Concept Design Phase Using
143
Safety Science 109 (2018) 130–143
STAMP/STPA. [Online]. Available: http://sunnyday.mit.edu/safer-world/.
Gerrit, B., Torben, S., Markus, M., 2017. Safety analysis based on systems theory applied
to an unmanned protective vehicle. In: 4th European STAMP Workshop 2016.
Harkleroad, E., Vela, A., Kuchar, J., 2013. Review of Systems-Theoretic Process Analysis
(STPA) Method and Results to Support NextGen Concept Assessment and Validation.
MIT Lincoln Laboratory, Washington, DC.
Haruka,N., Masa, K., Yuko, M., Nancy, L., 2011. Safety Guided Design of Crew Return
Vehicle in Concept Design Phase using STAMP/STPA. [Online]. Available: http://
sunnyday.mit.edu/safer-world/CRV.pdf.
Kleve, G.R., 2017. How safe is our Nurse call system? In: 4th European STAMP Workshop
2016.
Lahoz, C.H.N., 2016. Systematic Review on STPA Final Results. In: STAMP Workshop:
MIT Partnership for a Systems Approach to Safety (PSAS), Massachusetts.
Leveson, N.G., 2011. Engineering a Safer World: Systems Thinking Applied to Safety,
Cambridge. MIT Press, Massachusetts.
Leveson, N.G. An STPA Primer Version1, August 2013. [Online]. Available: http://
sunnyday.mit.edu/STPA-Primer-v0.pdf.
Lopes, G.D., Hirata, C.M., Melo, B.J.d., 2015. A Rule-based Approach for Safety Analysis
using STAMP/STPA. In: 2015 1st URSI Atlantic Radio Science Conference (URSI AT-
RASC).
Maria, M.C., Karanikas, N., Anastasios, P., 2017. Application of STPA on small drone
operations: a benchmarking aproach. In: 4th European STAMP Workshop 2016.
Petit, R., 2004. Lessons learned applying UML in embedded software systems designs. In:
Second IEEE Workshop on Software Technologies for Future Embedded and
Ubiquitous Systems.
Rejzek,M., Krauss, S.,Hilbes, C., 2015. MIT partnership for a systems approach to safety
(PSAS). [Online]. Available: < http://psas.scripts.mit.edu/home/wp-content/
uploads/2015/03/2015-Martin-SafetyDrivenDesign_UML_STPA.pdf>.
Robert,A., Mihhail, F., Floris, G., Pentti, K., Are,P., 2017. Systems-theoretic process
analysis of maritime traffic safety management in the Gulf of Finland (Baltic Sea). In:
4th European STAMP Workshop.
Shuai, Y., Deming, Z., 2014. Research on methodology for safety generation and ver-
ification. In: International Conference on Mechatronic Sciences, Electric Engineering
and Computer (MEC), Shengyang, China.
Stringfellow, M.V., Leveson, N.G., Owens, B.G., Safety-Driven Design for Software-
Intensive Aerospace and Automotive Systems, Proceedings of the IEEE (Volume: 98,
Issue: 4, April 2010), pp. 515–525, 29 March 2010.
Sulaman, S.M., Abbas,T., Wnuk, K., Höst,M., 2014. Hazard analysis of collision avoidance
system using STPA. In: 11th International Conference on Information Systems for
Crisis Response and Management.
Sun, R., Zhong, D., 2013. Using STPA thinking to help convert Natural Language into
Finite Automaton. [Online]. Available: < http://psas.scripts.mit.edu/home/wp-
content/uploads/2013/04/01_Sun_Zhong_Using_STPA_convert_natural_language_
finite_automaton.pdf>.
Suo, D., An, J., Zhu, J.,2011. A new approach to improve safety of reconfiguration in
integgrated modular avionics. In: Digital Avionics Systems Conference (DASC), 2011
IEEE/AIAA 30th, Seattle, WA, USA, 2011.
Takuto, I., Leveson, N., Cody, F., Yuko, M., Haruka, N., 2011. Multiple Controller
Contributions to Hazards. [Online]. Available: http://sunnyday.mit.edu/safer-
world/.