This article has been accepted for publication in a future issue of this journal, but has not been
fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2968480, IEEE
Transactions on Industrial Informatics
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS
Design-to-Test Approach for Programmable
Controllers in Safety-Critical Automation Systems
Canlong Ma, Julien Provost, Member, IEEE,
Abstract—This paper presents a design-to-test approach for
enhancing black-box complete conformance testing of pro-
grammable controllers, where their specifications can be modeled
as finite state machines. Given an automation system, the testing
objective is to check whether its implemented controllers conform
to the expected behavior with regard to the specification models.
The design-to-test approach analyzes the specification models and,
if necessary, automatically modifies them at the cost of limited
design overhead, so that the testability of the final implemented
controllers can be ensured/improved. By design, this approach
also guarantees that the behavior of implemented controllers
remains unchanged during normal execution (i.e., when not
connected to a test bench).
Index Terms—design-to-test, programmable controller, single-
input-change, finite state machine, black-box testing
I. I NTRODUCTION
I NDUSTRIAL automation systems are essential bases of
modern life, where machineries and processes are con-
trolled by controllers through sensors and actuators, instead of
directly by human beings. Safety-critical systems are systems
whose failure may result in injury, loss of life or serious
environmental damage [1]. Thus, safety-critical automation
systems, or safety-critical parts of such systems, are required
to be assured completely, i.e., all possible behavior of a
system/part in all possible situations should be covered.
In this paper, the target objects are programmable con-
trollers, which are widely used in industrial fields such as man-
ufacturing and power plant. In the development and application
of programmable controllers, the fact that requirements are
becoming more complicated, highly individualized, and fre-
quently changed, poses great challenges not only to functional
development but also to verification and validation [2] [3].
As an important validation method, the use of testing is
strongly recommended by international safety standards such
as IEC 61508, ISO 26262 and IEC 61513 to be executed on
top of (formal or semi-formal) verification techniques. In a
traditional workflow, testing is usually only considered after
the design phase has been finished. However, a system that
has a good design does not necessarily have good testability.
It may require very high extra overhead when a system needs
to be re-designed and re-implemented because of deficient
testability.
Canlong Ma and Julien Provost were with Assistant Professorship
for Safe Embedded Systems, Technical University of Munich,
Garching bei München, Germany; currently, they are with
Validas AG, Munich, Germany. canlong.ma@tum.de,
ma@validas.de; julien.provost@tum.de,
julien.provost@validas.de
Manuscript received February 05, 2019; accepted January 04, 2020.
1551-3203 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 11,2020 at 12:05:12 UTC from IEEE Xplore. Restrictions apply.
1
In order to enhance the testability of programmable con-
trollers in safety-critical systems where complete testing is
required, a design-to-test (DTT) approach is presented in
this paper. In contrast to the traditional workflow, the DTT
approach considers testing much earlier in the development
process, i.e., during the specification design. After the func-
tional specification being modeled as finite state machines,
the models are checked with regard to their testability and,
if necessary, modified by the DTT approach. As a result, the
DTT approach improves the functional testability with limited
design overhead, reduces the testing overhead, and keeps the
controller behavior unchanged during normal execution.
A finalized version of the DTT approach is presented in
this paper, while early and intermediate results have been
published in two IEEE conferences in 2015 [4] and 2016 [5].
The main contributions of this paper are: modified mathemat-
ical formalism, improved algorithms, and a new case study.
Compared to previous results, the main improvements are: the
DTT approach now requires less design overhead to reach the
same or better testability, and it achieves better scalability for
large-scale applications.
The paper is structured as follows: Section II and sec-
tion III provide related work of the DTT approach and the
mathematical background, respectively. Section IV provides
an overview of the test objective and the test generation.
Section V presents the concrete issues encountered in testing,
and the corresponding methods with algorithms. A large-scale
case study is used to illustrated the application in section VI.
Finally, a discussion of this work is given in the last section.
II. R ELATED W ORK
A. Model-based testing
Model-based testing (MBT) is a variant of testings, where
test cases are generated from behavior models of system
specifications, and then executed on the implementations (such
as programmable controllers in this paper) [6].
MBT has attracted much research interest. One main stream
is formalization of the models that are used for testing pur-
poses [3]. For example, in [7] a UML-based development
process for IEC 61499 applications with automatic test case
generation is presented. In [8], the authors transformed Func-
tion Block Diagram programs into timed automaton models
to generate the test cases automatically. In [9], the authors
modeled Program Organization Unit oriented unit tests with a
SysML modeling approach for IEC 61131-3 control softwares.
The DTT approach presented in this paper also uses a formal
modeling language to model the system specification. Its
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2968480, IEEE
Transactions on Industrial Informatics
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS
methodology can be adapted and extended to other formalisms
as well.
B. Complete conformance testing
As a type of model-based testing, conformance testing
compares the observed behavior of an implementation, e.g.,
a programmable controller, to the expected behavior of the
specification models [10].
With complete conformance testing (CCT), the test genera-
tion considers all possible combinations of input signals from
all states. It is highly advantageous for safety-critical systems
because the whole system behavior is covered by the tests.
However, this characteristic also restricts its applicability to
large-scale systems, because the test case number grows ex-
ponentially with the input number. To cope with this problem,
one main direction is to generate a small set of test cases that
has high code coverage [11] [12]. Nevertheless, as concluded
in [13], tests with coverage criteria satisfaction cannot always
effectively find faults. [14] and [15] made efforts in another
direction: the former generated a small set of test case that
guarantees full coverage of nominal behavior of the controller;
the latter inserted a selected set of faults into set of test cases
based on expertise, to test certain error handling routines.
However, with both methods, no guarantee can be achieved
that all critical faults are covered by their tests.
As a result, complete conformance testing is still essential to
safety-critical systems/parts, which is supported and enhanced
by the DTT approach in this paper.
C. Inspiration of DTT
The concept of DTT was inspired from a DFT (design-for-
test) technique, which adds testability features in the design of
integrated circuits (ICs) [16]. By considering testability early
in the design, users are able to lower power consumption, re-
duce test time, and improve fault coverage in testing [17] [18].
A similar idea has also been found in the development of
automotive applications [19], where a fast functional safety
verification method is applied during early design phase, so
that performance of software reliability and response time can
be optimized in later phase of development.
The DTT approach also considers testing early in the
design of specification models, so that better testability can
be achieved later in the testing of programmable controllers.
D. Software instrumentation
The idea of modifying the specification was inspired from
software instrumentation techniques. From instrumented pro-
grams, expected data traces can be produced and used to
diagnose misbehaviors in the embedded systems [20] [21].
The difference is yet obvious, instrumentation techniques
manipulate the specifications directly on the code level, while
the DTT approach modifies the specifications on the model
level. With the DTT approach, code is automatically generated
from the modified specification models. Therefore, additional
quality assurance measures that are required for the instru-
mented code, such as model checking and code inspection,
are not required in the application of the DTT approach.
1551-3203 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 11,2020 at 12:05:12 UTC from IEEE Xplore. Restrictions apply.
2
III. M ATHEMATICAL BACKGROUND
To apply the DTT approach, system specification is modeled
as a set of communicating Moore machines extended with
Boolean signals1 . Actually, the approach can also be applied
to other signal-based models (such as Grafcet) with stability
search semantics, where all possible paths of each transition
are considered independently of the evaluation/execution or-
der [22].
A. Communicating Moore machine extended with Boolean
signals
A communicating Moore machine extended with Boolean
signals is defined by an 8-tuple (L, linit , I, C, O, Gδ , δ, λ),
where:
• L is a finite set of locations. A location represents a logic
state of a single model for a subsystem/component2 .
• linit is the initial location, linit ∈ L.
• I is a finite set of Boolean input signals.
• C is a finite set of internal Boolean communicating
variables that are related to locations; a communicating
variable is denoted as ‘X(location)’, e.g., ‘X(l1 )’.
• O is a finite set of Boolean output signals/output actions.
• Gδ := expr(I, C) is a finite set of transition guards,
which are Boolean expressions3 built up by inputs and
internal variables.
• δ : L × Gδ → L is the transition function that maps the
current location and transition guard to the next location;
a transition is fired when its source location is active and
its guard is evaluated as True; ‘∆’ is used to denote the
set of ‘δ’.
O
• λ : L → 2 is the output function that maps the locations
to their corresponding output signals; ‘Λ’ is used to
denote the set of ‘λ’.
The models are also presented in graphical form. As exam-
ples, two models are presented in Fig. 1.
Model 1 i1 ¬i1 ∧ ¬i2
∅ o1 ∅
X(l5 ) ∧ i1
l1 l2 l3
Model 2 ¬i1 ∧ ¬i2 ∧ ¬i3 i1 ∧ ¬i2 ∧ i3
o2 ∅ o2
¬i1 ∧ i2 ∧ ¬i3 i1 ∧ i2 ∧ i3
l5 l4 l6
Fig. 1. Example: Two Moore machine models with Boolean signals
A location l is drawn as a rounded rectangle or a circle. It
has either no observable output, e.g., ∅ in l1 , or an externally
observable output4 , e.g., o1 in l2 . A transition δ is represented
by an oriented arc with a guard g(δ), e.g., ¬i1 ∧ ¬i2 for the
transition from l2 to l3 . Internal communicating variable can
also be used in building transition guards. For instance, X(l5 )
1A Boolean signal can be assigned the value ‘1’ (True) or ‘0’ (False).
2 The term state is used explicitly to represent a state in a composed model
(introduced in the next subsection), which does not contain any internal
communicating variable; while the term location is used for an individual
model. For an individual model alone, its state is defined by L × C.
3 Boolean operators used in this paper: ∧: AND; ∨: OR; ¬: Negation.
4 For readability reasons, in this model, only active outputs are presented,
e.g., in l5 , o2 implicitly means o2 ∧ ¬o1 .
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2968480, IEEE
Transactions on Industrial Informatics
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 3

is assigned the value ‘1’ when l5 is activated; if i1 has the
value ‘1’, and at the same time if l3 is active, the transition
from l3 to l2 can be fired.
B. Stabilized composed automaton
After the specification is modeled as individual Moore
machines, the models are then composed in parallel to build
a monolithic model. For more information about this parallel
composition operation, readers can refer to [22] and [23] (page
79).
As introduced earlier in this section, the stability search
semantics is used in the composition. Given a set of individual
models that can run in parallel, all the transitions from all the
active locations are evaluated, and executed if they are enabled;
the firing of transitions continues until none of them can be
fired anymore, without changing the values of inputs. Thus, a
stable situation is reached. Thereby, the obtained monolithic
model is named a Stabilized Composed Automaton (SCA). A
software program proposed in [22], Teloco, is used in this
composing process.
An SCA is defined by a 7-tuple (S, sinit , I, O, Ge , e, λs ),
where:
• S is a finite set of states. A state represents a combination
of locations from all the individual models.
• sinit is the initial state, sinit ∈ S.
• I is a finite set of Boolean input signals (same as used
in the individual models).
• O is a finite set of Boolean output signals/output actions
(same as used in the individual models).
• Ge := expr(I) is a finite set of evolution guards, which
are Boolean expressions built up by inputs.
• e : S × Ge → S is the evolution function that maps
the current state and evolution guard to the next state;
an evolution can be understood as a transition between
states.
O
• λs : S → 2 is the output function that maps the states
to their corresponding outputs.
As an example, the SCA of the two models in Fig. 1 is
presented in Fig. 2. For example, s4 is a state, which represents
a combination of l2 and l6 , and it has the output from the both
locations, i.e., o1 and o2 . In the model, all the states are stable,
which means: without changing the input values, no evolution
will be fired, and an active state will remain active.
¬i1 ∧ i2 ∧ ¬i3
o2
∅
i1 ∧
¬i1 ∧¬i2 ∧¬i3
s1 (l1 , l4 )
¬i2 ∧ i3
i1 ∧ (i2 ∨ ¬i3 )
IV. T EST OBJECTIVE AND TEST GENERATION
A. Test objective
A programmable controller can be programmed with func-
tions and instructions. It executes in a cyclic mode which ful-
fills the so-called hard real-time requirement. In each cycle, a
controller runs successively: input reading, program execution,
and output updating.
In the black-box conformance testing of a programmable
controller, the internal structure of the implementation is not
visible, while its expected behavior (from its specification
models) is known to the testers.
As a basic test case, a test unit for an evolution (e.g., the
evolution from s2 to s4 in Fig. 2) consists of three phases [24]:
1) Before testing: to activate the expected source state (e.g.,
s2 ) from the current active state by inputting an auxiliary
sequence of signals (e.g., from s3 to s2 )
2) During testing: to feed the testing input signals to the
controller (e.g., {101} in the order of (i1 , i2 , i3 ))
3) After testing: to determine the destination state by com-
paring the observed output signals to the expected out-
puts; if the observed output is not unique, to apply another
auxiliary sequence of signals (e.g. s4 and s6 have the
same outputs o1 ∧ o2 )
During the three phases, three issues have been identified:
the observability issue, the controllability issue and the single-
input-change-testability (SIC-testability) issue. More details
about them are presented in the next section.
B. Test generation
With a model-based approach, test cases and sequences are
generated from specification models.
Firstly, system specification is modeled (manually) as indi-
vidual models, which are then composed into an SCA. Next,
from the SCA, an equivalent Mealy machine model is built by
explicitly representing all Boolean conditions of all evolutions
over the Boolean input set. Apart from evolutions between
states, the Mealy machine contains also self-loop evolutions
which are omitted in the graphical representation of the SCA.
Finally, a test sequence is constructed by traversing all the
evolutions from all the states in the Mealy machine. A test is
considered complete, when its test cases are generated from
the complete specification. This test generation method and
its implemented tool Teloco have been presented in details
in [22].
The DTT approach, as an enhancement of complete testing,
s3 (l1 , l5 )
uses the tool Teloco for generating SCA and Mealy machine.
i1

i1 ∧ i2 ∧ i3 ¬i1 ∧ i2 ∧ ¬i3 V. T HE DTT APPROACH

o1 ∧ o2 o1 o1 ∧ o2
i1 ∧ ¬i2 ∧ i3 A. SIC-testability & T-guard method
s4 (l2 , l6 ) s2 (l2 , l4 ) s6 (l2 , l5 )
¬i1 ∧
¬i1 ∧ ¬i2 ¬i1 ∧ ¬i2 ¬i1 ∧ i1
¬i2 ∧ i3
The concept of single-input-change (SIC) and multiple-
¬i2 ∧ ¬i3
input-change (MIC) were firstly used in testing activities of
i1 ∧ i2 ∧ i3 ¬i1 ∧ i2 ∧ ¬i3
o2 ∅ o2 electronic circuits. Compared to MICs, SICs are advantageous
i1 ∧ ¬i2 ∧ i3 ¬i1 ∧¬i2 ∧¬i3
s7 (l3 , l6 ) s5 (l3 , l4 ) s8 (l3 , l5 ) in efficiently addressing decoder faults, requiring less power
consumption, and reaching higher fault coverage [25].
Fig. 2. The SCA of the two models in Fig. 1 In the testing of programmable controllers, SICs are also
preferred over MICs. But the reason is different: when multiple

1551-3203 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 11,2020 at 12:05:12 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2968480, IEEE
Transactions on Industrial Informatics
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 4

input signals simultaneously change their values, the actual MIC, i.e., ¬i1 ∧ i2 ∧ ¬i3 to i1 ∧ ¬i2 ∧ ¬i3 , can be regarded
signal values percepted by a controller may deviate from the as a SIC, and therefore s3 becomes fully SIC-testable.
expected values that the controller is supposed to read [26]. A generalized T-guard method is depicted in Alg. 1. In brief,
The physical causes are: in order to obtain full SIC-testability, a minimum set of T-
1) In natural world, two events, i.e., changes of two different guards are added to the individual specification models (not
physical signals, cannot occur at the same time exactly. only to the SCA).
2) Due to the cyclic execution mode, a programmable con-
troller reads the input values only in the first cycle phase. Algorithm 1: Pseudo-code of the T-guard method
If an input changes its value just after the phase, the new Input: L, ∆, Gδ , I
value can only be read in the next cycle. Result: GSIC
1 Initialization: IN SIC := ∅; Ttarget := ∅;
Experiments have proven that the occurrence of this issue 2 LT := ∅; GSIC := Gδ ;
cannot be neglected, especially for large-scale systems which 3 begin
4 GE−N SIC := T eloco(L, ∆, I);
contain multiple I/O cards [27]. To cope with this issue, some 5 foreach gE−N SIC ∈ GE−N SIC and i ∈ I do
researchers try to build test sequences that consist of as many 6 IN SIC += {i | i ∧ gE−N SIC = gE−N SIC };
SIC steps as possible [26] [28]. Therefore, the issue is named 7 IN SIC += {¬i | ¬i ∧ gE−N SIC = gE−N SIC };
/* check if i or ¬i is an element in gE−N SIC
as SIC-testability issue. */
However, it has been proven not always possible to build a 8 IN SIC := sort(IN SIC , GE−N SIC );
test sequence that contains only SIC test steps for a practical 9 foreach i ∈ IN SIC do
10 i.value := F alse;
system. As a part of the DTT approach, a T-guard method is /* set the initial values to False */
presented to provide a thorough solution to this issue. 11 foreach j ∈ IN SIC do
The two models presented in Fig. 1, and their SCA (Fig. 2) 12 j.value
_:= T rue;
are used in the following to illustrate the SIC-testability issue 13 if gE−N SIC 6= F alse then
and the T-guard method in details. GE−N SIC
14 Ttarget += {j};
Using the DTT approach, the SIC-testability result is calcu- 15 j.value := F alse;
lated by Teloco [22]. Due to its simplicity, the SIC-testability /* this means j is an essential element in
of the system in Fig. 1 can also be identified manually from Ttarget , a minimum set of IN SIC */
the SCA in Fig. 2. For instance, the non-SIC-testable part of 16 foreach gδ ∈ Gδ and l × gδ → l0 do
s3 is i1 ∧ ¬i2 ∧ ¬i3 , which is actually a guard of an evolution 17 foreach k ∈ Ttarget do
18 if k ∧ gδ = gδ then
from s3 to s6 in the Mealy machine. Obviously, it cannot 19 LT += {l};
be obtained with a SIC from the guard of the only incoming /* l is a location that has outgoing
evolution of s3 , i.e., ¬i1 ∧ i2 ∧ ¬i3 ; and all its SIC-“siblings” transitions that have inputs in
Ttarget */
will make either ¬i1 ∧¬i2 ∧¬i3 or i1 evaluated as True, which
enables an outgoing evolution to either s1 or s6 . Therefore, 20 foreach l ∈ LT and gδ ∈ Gδ do
s3 is not fully SIC-testable. 21 if ∃{l, gδ } | l × gδ → l0 then
22 GSIC −= {gδ };
To solve this issue, the T-guard method transforms all MICs 23 gδ := gδ ∧ T g;
into SICs with an extra non-functional input signal, which /* add a T-guard to the guards of outgoing
is named as T-guard, denoted as ‘T g’. As an example, a transitions of l */
24 GSIC += {gδ };
schematic diagram of signal and state changes before/after
adding a T-guard for the non-SIC-testable evolution of s3 is
presented in Fig. 3.
The inputs of Alg. 1, I, L, ∆ and Gδ , represent respectively
the full set of inputs, the unions of (with regard to all the
Controller I E O I E O I E O I E O I E O I E O I E O I individual models): the full sets of locations, the full sets of
Cycle
transitions, and the full sets of transition guards.
Tg 1
0 Firstly, GE−N SIC , a non-SIC-testable subset of the evolu-
i1 1
0 tion guards for all states, is obtained by running Teloco [22]
i2 10 (line 4 in Alg. 1). For the example in Fig. 2, GE−N SIC is
active state s3 s6 {i1 ∧ i2 ∧ ¬i3 , i1 ∧ ¬i2 ∧ ¬i3 }.
Then, all the inputs (considering both themselves and their
time
negations) that are involved in the evolution guards belonging
Fig. 3. Changes of signals and states after adding a T-guard to GE−N SIC are picked out and listed in IN SIC (lines 5 to
7). After that, according to the appearance frequency of its
Before a MIC is executed, the input signal T g is set to the elements in GE−N SIC , IN SIC is sorted in ascending order
value ‘0’, so all the outgoing and self-loop evolutions from (lines 8). The next step is to filter out the elements IN SIC
the current state, i.e., s3 , are frozen from being fired. After iteratively, in order to obtain Ttarget , a minimum set of IN SIC .
the MIC is stabilized, i.e., i1 and i2 have finished changing The criterion of the filtering is that: as long as all the inputs in
their values, T g is set back to the value ‘1’. Then, only the Ttarget are protected by T-guards, all the guards in GE−N SIC
expected evolution will be fired. In other words, a previous are guaranteed SIC-testable (lines 9 to 15).

1551-3203 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 11,2020 at 12:05:12 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2968480, IEEE
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS
Compared to the previous algorithm, the newly added
sorting step helps to further improve the filtering effect, and
therefore may obtain a further shrunk Ttarget . For the example
in Fig. 2, IN SIC is {i2 , ¬i2 , i1 , ¬i3 }, Ttarget is {¬i3 }.
Now the focus is moved from the SCA to the individual
models. All the transition guards that contain non-SIC-testable
inputs should be protected with T-guards. Besides, to pre-
vent an unexpected evolution/transition from being fired, if
a location has any outgoing transition containing a non-SIC-
testable input, the guards of all the outgoing transitions from
this location should be protected by T-guards (lines 16 to 19).
Finally, as output of Alg. 1, GSIC is obtained through
updating all the relevant transition guards by adding a T-guard
to them (line 20 to 24). For the example in Fig. 1, guards for
the transitions l4 to l5 and l5 to l4 are directly updated, and
the guard for the transition l4 to l6 is indirectly updated. The
updated models are presented in Fig. 4.
Model 1 i1
∅ o1
l1 l2
¬i1 ∧¬i2 ∧¬i3 ∧T g i1 ∧ ¬i2 ∧ i3 ∧T g
Model 2
o2 ∅ o2
¬i1 ∧ i2 ∧ ¬i3 ∧T g
l5 l4
Fig. 4. The models in Fig. 1 updated with T-guards
B. Observability & O-action method
As introduced in section IV-A, after testing, there are two
methods to identify/distinguish the currently active state:
1) By observing the current outputs directly
2) By applying an distinguishing sequence [24]
A strong hypothesis is required to use the first method:
all the states must have unique observable output actions. It
is however often not fulfilled by real systems. The simple
example in Fig. 2 does not fulfill this hypothesis either.
The second method is more generally applicable. For the
example in Fig. 2, both s4 and s6 have the outputs o1 ∧o2 ; then
a tester can apply {111}, if observing o1 , then the destination
state was s4 , the right one, the test passes; if observing o1 ∧o2 ,
then the destination state was s6 , a wrong one, the test fails.
However, as stated in [24], such sequences do not always
exist; and if they exist, their lengths might grow exponentially
to the number of states in a system, which can cost huge
testing overhead for large-scale systems.
This problem is named as observability issue. The O-action
method is presented, as a part of the DTT approach, to solve
this issue. The example in Fig. 1 and Fig. 2 is used again in
the illustration.
The Observability result is calculated by the DTT approach
automatically. But for this simple example, it can also be
manually identified from the SCA in Fig. 2: s1 and s5 have
the same output ∅, s4 and s6 have the same output o1 ∧ o2 ,
s3 , s7 and s8 have the same output o2 . Obviously, the SCA
is not (directly) fully observable.
Alg. 2 depicts a generalized method to obtain full observ-
ability by adding a minimum set of O-actions, which are extra
non-functional output signals.
1551-3203 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 11,2020 at 12:05:12 UTC from IEEE Xplore. Restrictions apply.
Transactions on Industrial Informatics
5
Algorithm 2: Pseudo-code of the O-action method
Input: S, L, Λ, Λs , #model
Result: ΛObs
1 Initialization: LN Obs := ∅; ΛObs := Λ; #OA := 0 ;
2 begin
3 foreach si ∈ S, sj ∈ S, si 6= sj do
4 if λs (si ) = λs (sj ) then
5 for n = 1 : #model do
6 if λn (li ) = λn (lj ),
7 li ∈ si , lj ∈ sj , li 6= lj then
8 LN Obs += {li , lj };
9 for n = 1 : #model do
10 LN Obs,n := LN Obs ∩ Ln ;
/* gather the non-observable locations that
are from Ln , the set of all the locations
from the nth model
*/
11 #OA,n := dlog2 |LN Obs,n | e;
/* the minimum number of O-actions that are
needed
h for a model i*/
12 OAn := oa#OA +1 , oa#OA +2 , · · · , oa#OA +#OA,n ;
/* OAn is a list of O-actions (oa) for one
model */
¬i1 ∧ ¬i2
∅ 13 #OA += #OA,n ;
X(l5 ) ∧ i1 /* the number of O-actions that are needed for
l3 all models in the whole system */
14 foreach l ∈ LN Obs,n do
15 ΛObs −= {λ(l)};
i1 ∧ i2 ∧ i3 16 λ(l) := λ(l) ∧ minterm(OAn );
l6 /* update the output by adding a minterm,
which is a unique combination of
elements in OAn */
17 ΛObs += {λ(l)};
Similar to Alg. 1, S, Λs , L, Λ are the inputs of Alg. 2, which
represent respectively the full set of states, the full set of
outputs of states, the unions of (with regard to all the individual
models): the full sets of locations, and the full sets of outputs
of locations. The last input, #model , is defined as the number
of individual models. For the example in Fig. 1, #model is 2.
Firstly, all the states are checked if they have same outputs
with other states (lines 3 to 4). If yes, the locations contained
in these states will be further examined. In an individual
model, two locations will be considered as the cause of the
observability issue of the states they are contained in, if the
locations have the same output action. These locations are
stored in LN Obs (lines 5 to 8). For the example in Fig. 1,
LN Obs contains 4 locations: l1 , l3 , l5 , l6 .
Then, the focus is moved to the individual models. The
observability issue is analyzed inside each individual model. A
list of n Boolean O-actions can express 2n different outputs5 .
Thus, a minimum number of O-actions that are needed for
an individual model to become fully observable is obtained
by calculating the logarithm of the number of elements in
LN Obs,n with the base of 2 (because of Boolean signals) (line
9 to 11)6 .
As output of Alg. 2, ΛObs is obtained through updating the
outputs of locations by inserting a unique combination of O-
action values to the relevant locations in LN Obs,n (lines 12 to
17). For the example in Fig. 1, each model has two locations in
5 De-activated outputs are actually equivalent to no appearance, e.g., oa ∧
1
¬oa2 is equivalent to oa1 .
6 The result is rounded up to an integer with the ceiling function.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2968480, IEEE
Transactions on Industrial Informatics
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 6

LN Obs,n , so one O-action is needed for each model. Therefore, Algorithm 3: Pseudo-code of the C-guard method
two O-actions are required to distinguish the total system. The Input: S, E, L, ∆, Gδ , #model , LimitCtr
updated models are presented in Fig. 57 . Result: ∆Ctr , GCtr
1 Initialization: ECtr := ∅; ∆Ctr := ∆; GCtr := Gδ ;
Compared to the previous algorithm, the new algorithm 2 begin
assigns exclusive O-actions to each relevant model instead of 3 foreach si ∈ S, sj ∈ S do
using shared O-actions for all models. As a result, the O- 4 if si = sj then
5 P athS (si , sj ) := 0;
action method improves the scalability of handling large-scale 6 else if ∃(si × ge → sj ) ∈ δL then
systems (especially distributed systems); when a subsystem 7 P athS (si , sj ) := 1;
or a part needs to be re-designed, the other subsystems/parts 8 else
9 P athS (si , sj ) := ∞;
are less affected. As a potential drawback, this might lead
to a larger total number of O-actions; nevertheless, users can 10 P athS := Floyd-Warshall(P athS );
11 while max(P athS
) > LimitCtr do
always select the result that fits best to their needs. 12 ∆Ctr , ∆new :=
EvoCalc(P athS , ∆Ctr , S, #model , L);
Model 1 i1 ¬i1 ∧ ¬i2 /* EvoCalc is presented in Alg. 4 */
∅ ∧oa1 o1 ∅ ∧¬oa1
X(l5 ) ∧ i1 13 #Ctr := |∆new |;
l1 l2 l3 /* #Ctr
n is the number of C-guards
o to be added */
Model 2 ¬i1 ∧ ¬i2 ∧ ¬i3 i1 ∧ ¬i2 ∧ i3 14 C := ctr1 , ctr2 , · · · , ctr#Ctr ;
o2 ∧oa2 ∅ o2 ∧¬oa2 /* C is a set of C-guards */
¬i1 ∧ i2 ∧ ¬i3 i1 ∧ i2 ∧ i3
l5 l4 l6 15 foreach δCtr ∈ ∆new do
16 gδ,Ctr := C(i);
/* gδ,Ctr is the guard for the δCtr created in
Fig. 5. The models in Fig. 1 updated with O-actions line 6 in Alg. 5; i is the index of δnew in
∆Ctr */
17 GCtr += {gδ,Ctr };
18 foreach gδ ∈ G∆ do
C. Controllability & C-guard method 19
0
if ∃ldes 0
| ldes × gδ → ldes then
As introduced in section IV-A, an expected state should be 20 GCtr −= {gδ };
21 gδ := gδ ∧ ¬C(i);
activated before testing. 22 GCtr += {gδ };
This process is actually a switch of the active status from
a random (current) state to another expected state, which
can be realized by applying a homing or a synchronizing
sequence [24]. For the example in Fig. 2, assuming the current
active state is s3 and the expected state is s2 , a homing states. It is initialized as follows: the path cost for a state to
sequence can be {000, 110}. itself is set to 0; the path cost for a pair of states is set to 1, if
For real systems, some states can be either not reachable there is a direct evolution between them; otherwise, the cost
from some other states, or reachable but over a long distance, is set to ∞ . Then, P athS is updated with the Floyd-Warshall
i.e., via very long sequences. The two questions, whether and algorithm, which calculates the indirect path costs for all pairs
how fast the switch of the active status can be performed, are of states (line 10).
the concerns of the so-called controllability issue. Tab. I presents the values of P athS for the example in
The C-guard method, as the last part of the DTT approach, Fig. 2. The values for the initial and updated path costs (for
aims at solving/easing the controllability issue by building the models before and after being updated with C-guards) are
connections among locations. To be more concrete, a mini- depicted in black and green, respectively.
mum set of C-guard transitions are added to the individual
models. C-guards are extra non-functional input signals. TABLE I
PATH COST MATRIX FOR THE SYSTEM IN F IG . 2
A generalized algorithm of the C-guard method is presented
in Alg. 3. Two functions of Alg. 3, EvoCalc and T ranCalc, From
s1 s2 s3 s4 s5 s6 s7 s8
are presented separately in Alg. 4 and Alg. 5. Again, the To
s1 0 ∞/2 1 ∞/3 ∞/1 ∞/2 ∞/2 ∞/2
example in Fig. 1 and Fig. 2 is used in the illustration. s2 1 0 2 1 ∞/2 ∞/3 ∞/3 ∞/3
As inputs of Alg. 3, S, E, L, ∆, Gδ , #model , LimitCtr rep- s3 1 ∞/3 0 ∞/4 ∞/2 ∞/3 ∞/3 ∞/3
resent respectively the full set of states, the full set of s4 1 1 2 0 ∞/2 ∞/3 ∞/3 ∞/3
evolutions, the unions of (with regard to all the individual s5 2 1 2 2 0 1 1 1
s6 2 1 1 2 2 0 3 1
models): the full sets of locations, the full sets of transitions, s7 2 2 3 1 1 2 0 2
and the full sets of transition guards. #model is again defined s8 3 2 2 3 1 1 2 0
as the number of individual models. The last input LimitCtr
is the expected controllability (i.e., path cost between states, The maximum of P athS is then compared to LimitCtr , the
a positive integer) to be specified by users. expected controllability: if it exceeds, then ∆new , a minimum
Firstly, the initial controllability is calculated (lines 3 to 9). set of C-guard transitions, is calculated for the individual
P athS is defined to be a path cost matrix for all pairs of models (lines 11 to 12). The guards of the newly created
7 For better display and clear illustration, not only active outputs but also de-
transitions are consequently assigned (lines 13 to 17). It is
activated O-actions are presented, e.g., in l3 , ∅ ∧ ¬oa1 is actually equivalent worth noting that, after the C-guard transitions have been
to ∅; besides, the T-guards are not presented in this figure. added, for stability reason, the negation of the C-guards should

1551-3203 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 11,2020 at 12:05:12 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2968480, IEEE
Transactions on Industrial Informatics
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 7

be added to the direct consecutive transitions of the newly Algorithm 4: Pseudo-code of EvoCalc
created C-guard transitions (lines 18 to 22). In the example, the Data: P athS , ∆Ctr , S, #model , L
maximum of initial path cost is ∞, which means some states Result: ∆Ctr , ∆new
1 Initialization: maxsum−r := 0; maxsum−c := 0;
are not reachable from some other states. Actually, this can 2 ∆new := ∅, minsum−r := ∞; minsum−c := ∞;
even be easily observed from the SCA model. After assigning 3 begin
an expected value to LimitCtr (e.g., 10), a C-guard transition 4 foreach si ∈ S, sj P
∈ S do
5 Dsum−si := sj P athS (si , sj );
has been added from l3 to l1 , the negation of the C-guard has
/* Sum of path
P costs between states in row */
been added to the guard of an existing transition from l1 to l2 . 6 Dsum−sj := si P athS (si , sj );
As a result, all the states in the SCA become reachable from /* Sum of path costs between states in column
other states, as presented in Tab. I (the updated data is marked */
7 if Dsum−si > maxsum−r then
in green). The updated models are presented in Fig. 68 . 8 maxsum−r := Dsum−si ; smax−r := si ;
9 if Dsum−si < minsum−r then
ctr1 10 minsum−r := Dsum−si ; smin−r := si ;

Model 1 ¬i1 ∧ ¬i2 11 if Dsum−sj > maxsum−c then

i1 ∧¬ctr1
∅ o1 ∅ 12 maxsum−c := Dsum−sj ; smax−c := sj ;
X(l5 ) ∧ i1
l1 l2 l3 13 if Dsum−sj < minsum − c then
¬i1 ∧ ¬i2 ∧ ¬i3 i1 ∧ ¬i2 ∧ i3
14 minsum−c := Dsum−sj ; smin−c := sj ;
Model 2
o2 ∅ o2
¬i1 ∧ i2 ∧ ¬i3 i1 ∧ i2 ∧ i3 15 Br := maxsum−r − minsum−r ;
l5 l4 l6
16 ∆new,r :=
T ranCalc(smin−r , smax−r , ∆Ctr , #model , L);
Fig. 6. The models in Fig. 1 updated with C-guards /* Calculate benefit and C-guards to be added in
row; T ranCalc is presented in Alg. 5 */
17 Bc := maxsum−c − minsum−c ;
The function EvoCalc (line 12 in Alg. 3) is presented in 18 ∆new,c := T ranCalc(smax−c , smin−c , ∆Ctr , #model , L);
Alg. 4. /* Calculate benefit and C-guards to be added in
column */
Firstly, the elements of P athS , i.e., P athS (si , sj ), are 19 if Br == Bc == ∞ then
summed in rows and columns (lines 4 and 6). The two 20 if |∆new,r | < |∆new,c |
maximums and two minimums of the sums, i.e., maxsum−r , /* |∆new | is the number of elements in ∆new */
21 then
minsum−r , maxsum−c , minsum−c , are obtained through it- 22 enew := e(smin−r , smax−r );
erative comparisons (lines 7 to 14). 23 ∆new += ∆new,r ;
Then, the benefits (reduced path costs) and expenses (num- 24 else
25 enew := e(smax−c , smin−c );
ber of added C-guards) of creating C-guard transitions ac- 26 ∆new += ∆new,c ;
cording to the results in rows (which means to build a new 27 else
connection from a “cheapest destination” state to a “most 28 if Br /|∆new,r | > Bc /|∆new,c | then
costly destination” state) and columns (which means to build 29 enew := e(smin−r , smax−r );
30 ∆new += ∆new,r ;
a new connection from a “most costly source” state to a 31 else
“cheapest source” state) are calculated and compared. If Br 32 enew := e(smax−c , smin−c );
and Bc are both ∞, that means a not fully reachable SCA 33 ∆new += ∆new,c ;
becomes fully reachable after adding C-guards according to 34 ∆Ctr += ∆new ;
either selection. In this case, which way that requires less C- 35 P athS := Floyd-Warshall(P athS , enew );
guards is a better solution. Otherwise, the selection is based on
the value of Br /|∆new,r |, i.e., the “price-performance ratio”
of the C-guards to be added (lines 20 to 34).
After a new evolution is added, the path costs for other D. Design, testing & normal execution
pairs of states can also be indirectly shortened, which are then This DTT approach has been implemented as an open-
re-calculated using the Floyd-Warshall algorithm (line 35). source MATLAB toolbox, which will be publicly available
Compared to the previous algorithm that only considers the at: http://www.ses.mw.tum.de/
most costly destination/source states in making the selection, In the design phase, the specification models are checked
the new algorithm permits to require less C-guards in achiev- and, if necessary, modified by the DTT approach.
ing the same or even better controllability.
In the testing, T-guards are set to the value ‘0’ or ‘1’
Alg. 5 presents the function T ranCalc (line 16 and 18 in
accordingly, so that all MICs can be stabilized and considered
Alg. 4). When considering building a new connection between
as SICs. O-actions are used to distinguish directly all the
a pair of states, Alg. 5 will traverse iteratively all the pairs of
states that are tested. C-guards are set to the value ‘0’ or ‘1’
locations that are: 1. contained in the two different states, 2.
accordingly to enable the control of evolutions.
contained in the same model; and check if there is already
When a controller is put into normal use, i.e., after the
a direct transition between them. Where there is no existing
testing is completed, the input signals T g and ctri will be
transition, a C-guard transition will be created.
respectively set to the values 1 and 0, e.g., by connecting them
8 For better display, the T-guards and O-actions are not presented in this to 3.3V/5V/24V, and 0V, respectively. Since g ∧ 1 = g and
figure. g ∧ 0 = 0, the T-guards and C-guards will be inhibited and

1551-3203 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 11,2020 at 12:05:12 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2968480, IEEE
Transactions on Industrial Informatics
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 8

Algorithm 5: Pseudo-code of T ranCalc (with their relative position remaining unchanged) but can pick
Data: ssrc , sdes , ∆, #model , L and place a workpiece independently. The robot uses its first
Result: ∆new arm to deliver the workpiece from the table to the press. After
1 Initialization:∆new := ∅;
2 begin
being forged by the press, the workpiece is delivered by the
3 for n = 1 : #model do robot’s second arm to the deposit belt. Then the workpiece
4 foreach lsrc ∈ Ln , ldes ∈ Ln do is delivered by the deposit belt to an end station, where it is
5 if lsrc ∈ ssrc and ldes ∈ sdes and lsrc 6= ldes and
@δCtr | δCtr : lsrc × gδ → ldes ∈ ∆ then
checked by a test unit whether the forging is successful. If
6 ∆new += {δCtr | δCtr : lsrc × gδ,Ctr → the workpiece passes the test, it will be output from the cell;
ldes }; otherwise, it will be picked up by the crane and delivered to
/* the expression of gδ,Ctr is assigned
in line 16 in Alg. 3 */
the feed belt for another forging process.
In the system specification, safety aspects also need to be
considered, such as collisions between the robot arms, the
table, and the press, collisions between workpieces on the feed
belt and the deposit belt, both of which has a capacity of two.
cannot affect the original transition behavior. The O-actions
are output signals and do not affect any system behavior at Feed belt X(ERTIdle )∧wp1 ∧T g

any time. wp1 ∧T g ¬wp1 ∧ wp2 ∧T g

F B Run
∅ ∧oa4 ∧ oa5 ∅ ∧oa4 ∧ ¬oa5
As a conclusion, behavior of a controller during normal ∧¬oa4 ∧ oa5
F BIdle F BRunning F BOutput
execution is not changed by the DTT approach with regard to
X(ERTIdle ) ∧
its specification. ¬wp1 ∧ wp3 ∧T g ¬wp1 ∧T g
F B Run
∧¬oa4 ∧ ¬oa5
F BT ransf erN oIn
VI. C ASE S TUDY Robot
A benchmark case study (Fig. 7) originally presented in [29] ∅ ∧oa1 ∧
X(ERTOutput )
∧¬ctr1 ¬wp3
R A1P ick ∅
is used in this paper to illustrate the DTT approach. ¬oa2 ∧ ¬oa3
RIdle RArm1P icking RArm1P icked

ctr1 X(PIdle )

A. System description ∅ ∧oa1 ∧ wp4

R A1P lace
¬loc3 ∧ loc4 R CCW ∧¬oa1 ∧
¬oa2 ∧ oa3 oa2 ∧ oa3
As presented in Fig. 7, a production cell (PC) consists of RArm1P laced RArm1P lacing RM oveF 1
X(POutput ) ∧ X(ERTOutput )
seven components: a stock (S), a feed belt (FB), an elevating RArm12P icking

rotary table (ERT), a robot (R), a press (P), a deposit belt (DB), R CW
loc3 ∧ ¬loc4 R A1P ick ∧
R A2P ick
¬wp3 ∧ ¬wp4 ∅ ∧oa1 ∧
oa2 ∧ oa3
and a crane (C) 9 . Apart from the synchronization needed for RM oveB1 
X(DBBadOutN oIn ) ∨ X(DBOutput ) ∨
RArm12P icked
X(POutput ) ∧ X(ERTOutput ) 
cooperation and safety, each component runs independently. X(DBIdle ) ∧ X(PIdle ) ∧ ¬wp5 ∧T g

∅ ∧oa1 ∧ wp4 ∧ wp5 R A1P lace ∧ ¬loc3 ∧ loc4 R CCW ∧¬oa1 ∧

oa2 ∧ ¬oa3 R A2P lace oa2 ∧ ¬oa3
wp6 wp5 RArm12P laced RArm12P lacing RM oveF 2
Output
loc2 Deposit belt loc4
Test Fig. 8. Specification models of the feed belt and robot
unit

Legend: Arm2 The specification is modeled with 7 Moore machines, and

Crane/robot each contains 2, 4, 4, 12, 5, 7, and 6 locations, respectively.
location detector Robot Press
wp4
The two models for the feed belt and the robot are presented in
Workpiece detector Fig. 8 as illustrative examples. Same as in the previous section,
Arm1 the initial models are depicted in black. The T-guards, O-
Crane wp3
actions and C-guards added by the DTT approach are depicted
wp2 in blue, purple, and green, respectively.
Input
Stock loc1 Feed belt loc3
Elevating
wp1 rotary table B. Application of the DTT approach
The SCA of the individual models contains 2694 stable
Fig. 7. A sketch of the production cell states and 135,680 evolutions.
Applying the DTT approach, the analysis shows that 2690
The PC system is modeled with 17 input and 17 output
states are not fully SIC-testable. Then, 26 T-guards are added
signals, as listed in Tab. II.
to the specification models to solve this issue. In the SCA,
Unprocessed workpieces are stored in the stock, and are fed
2658 states do not have unique output actions. After adding 10
to the feed belt one by one. A workpiece is delivered through
O-actions to the models (respectively 0, 2, 1, 3, 0, 3, and 1 O-
the feed belt to the elevating rotary table, which can lift the
actions to the individual models), all the states become directly
workpiece to be picked up by the robot. The robot has two
distinguishable. The initial controllability of this system is ∞,
orthogonal arms, which are designed to rotate simultaneously
which means that some states are not reachable from some
9 The initials are used to represent the components in the specification other states. Added with 3 C-guards in the individual models,
models. all the states become reachable from all the other states, more

1551-3203 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 11,2020 at 12:05:12 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2968480, IEEE
Transactions on Industrial Informatics
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 9

TABLE II
I NPUTS & OUTPUTS OF THE PRODUCTION CELL

Input Description
wp1 / wp2 / wp3 / wp4 / wp5 / wp6 activated when a workpiece is detected by a corresponding sensor
loc1 / loc2 / loc3 / loc4 activated when the crane or the robot reaches a corresponding horizontal position
p high / p middle / p low activated when the press reaches a corresponding vertical position
ert high / ert low activated when the elevating rotary table reaches a corresponding vertical position
test pass / test fail activated when the workpiece passes or fails the test executed by the test unit

Output Description
S Feed the stock feeds a workpiece to feed belt
FB Run / DB Run the feed belt / deposit belt runs (in a forward direction)
ERT Rise / ERT Fall the elevating rotary table rises or falls
R CW / R CCW the robot rotates in a clockwise or counterclockwise direction
R A1Pick / R A1Place / R A2Pick / R A2Place the robot picks or places a workpiece with its first or second arm
P Rise / P Fall the press rises or falls
C MoveFWD / C MoveBWD the crane moves in a forward or backward direction
C Pick / C Place the crane picks or places a workpiece

TABLE III
E VALUATION OF SYSTEM TESTABILITY BEFORE AND AFTER USING THE DTT APPROACH

Specification models SIC-Testability Observability Controllability

for test generation Status #T-guard #Code change Status #O-action (total/individual) #Code change #Step #C-guard #Code change
Original models not fulfilled – – not fulfilled – – ∞ – –

With DTT Approach

fulfilled 30 31 fulfilled 5 / {0, 5, 5, 5, 5, 5, 5} 10 11 5 10
(previous algorithms)
With DTT Approach
fulfilled 26 27 fulfilled 10 / {0, 2, 1, 1, 2, 3, 1} 20 11 4 8
(new algorithms)

precisely speaking, within 13 steps. Furthermore, after adding
more C-guards, the controllability can be further improved,
e.g., the controllability of 11 steps can be achieved with 4
C-guards.
C. Influences over the executable code
Further analysis has been done to investigate the influences
of the DTT approach over the executable code of the con-
troller. PLC Structured Text (ST) code has been generated for
the individual models of the case study automatically via the
tool Teloco [22].
The code for the initial models contains 180 lines. The code
is increased by 1 line and modified in 26 lines after adding
the 26 T-guards, increased by 20 lines after adding the 10
O-actions, increased by 6 lines after adding the 3 C-guards.
As examples, following are some lines of the ST code
related to the T-guards, O-action and C-guards:
• tF B Id3 := X(F B Idle) AND wp1 AND T g;
• oa 4 := X(F B Idle) OR X(F B Output);
• tR M ov1 := X(R M oveB1) AND ctr 1;
As a result, it can be concluded that the increased overhead
for adding/modifying the code is linear to the number of the
inserted T-guards, O-actions, and C-guards.
D. Comparison of system testability on the case study
A quantitative evaluation of the system testability for the
case study, before and after using the DTT approach, is
presented in Tab. III. The results include SIC-testability, ob-
servability and controllability, as well as the lines of executable
ST code. The result of the previous algorithms is also provided
as a reference.
In summary, compared to the previous algorithms, the new
algorithms require less T-guards, less C-guards, and less O-
actions in the individual models (though more O-actions in
total) in achieving full SIC-testability, full observability, and
better controllability of the case study.
VII. C ONCLUSIONS AND DISCUSSIONS
This paper has presented a design-to-test approach for
enhancing complete testing of programmable controllers in
safety-critical automation systems. The approach aims to im-
prove the testability and reduce the testing overhead with
limited design overhead.
Firstly, system specification is modeled as Moore machines
extended with Boolean signals. The models are automatically
checked and, if necessary, modified by the DTT approach
so that they are ensured to achieve full SIC-testability, full
observability and better controllability. The application and
benefits of the DTT approach have been illustrated on a
benchmark case study.
It is worth underlining that all the T-guards and C-guards
added by the DTT approach can be inhibited in normal
execution, and all the O-actions are output signals that do not
influence the transition functions. Thus, the controller behavior
will remain strictly the same in its normal execution.
The DTT approach presented in this paper deals only with
discrete systems, but its methodology should be more gener-
ally applicable. In the future, an extension to hybrid systems
with continuous dynamics would be of interest. Besides, the

1551-3203 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 11,2020 at 12:05:12 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.2968480, IEEE
Transactions on Industrial Informatics
IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS
approach composes all individual specification models into an
monolithic SCA, which hinders its application to very large-
scale systems, because the SCA size grows exponentially with
the input number in a system. For future work, the monolithic
composition is expected to be replaced by modular methods
to enhance the scalability of the whole process.
R EFERENCES
[1] I. Sommerville, Software Engineering, 8th ed. Pearson Education
Limited, 2007.
[2] V. Vyatkin, “Software engineering in industrial automation: State-of-the-
art review,” IEEE Transactions on Industrial Informatics, vol. 9, no. 3,
pp. 1234–1249, aug 2013.
[3] G. Frey and L. Litz, “Formal methods in PLC programming,” in IEEE
International Conference on Systems, Man, and Cybernetics, vol. 4,
2000, pp. 2431–2436.
[4] C. Ma and J. Provost, “Design-to-test approach for black-box testing
of programmable controllers,” in IEEE International Conference on
Automation Science and Engineering (CASE), 2015, pp. 1018–1024.
[5] ——, “DTT-MAT: A software toolbox of design-to-test approach for
testing programmable controllers,” in IEEE International Conference on
Automation Science and Engineering (CASE), Fort Worth, Texas, USA,
2016, pp. 878–884.
[6] M. Utting, A. Pretschner, and B. Legeard, “A taxonomy of model-
based testing approaches,” Software Testing, Verification and Reliability,
vol. 22, no. 5, pp. 297–312, aug 2012.
[7] T. Hussain and G. Frey, “UML-based development process for IEC
61499 with automatic test-case generation,” in IEEE International
Conference on Emerging Technologies and Factory Automation, ETFA,
2006, pp. 1277–1284.
[8] E. P. Enoiu, D. Sundmark, and P. Pettersson, “Model-based test suite
generation for function block diagrams using the UPPAAL model
checker,” in IEEE 6th Int. Conf. on Software Testing, Verification and
Validation Workshops, 2013, pp. 158–167.
[9] M. Jamro, “SysML modeling of POU-oriented unit tests for IEC 61131-
3 control software,” in 19th International Conference on Methods and
Models in Automation and Robotics (MMAR). IEEE, 2014, pp. 82–87.
[10] A. Guignard, J.-M. Faure, and G. Faraut, “Model-based testing of PLC
programs with appropriate conformance relations,” IEEE Transactions
on Industrial Informatics, vol. 14, no. 1, pp. 350–359, 2018.
[11] D. Bohlender, H. Simon, N. Friedrich, S. Kowalewski, and S. Hauck-
Stattelmann, “Concolic test generation for PLC programs using coverage
metrics,” in Discrete Event Systems (WODES), 13th International Work-
shop on. IEEE., 2016, pp. 432–437.
[12] S. Ulewicz and B. Vogel-Heuser, “Increasing system test coverage in
production automation systems,” Control Engineering Practice, vol. 73,
pp. 171–185, 2018.
[13] G. Gay, M. Staats, M. Whalen, and M. P. Heimdahl, “The risks of
coverage-directed test case generation,” IEEE Transactions on Software
Engineering, vol. 41, no. 8, pp. 803–819, 2015.
[14] C. Ma and J. Provost, “Introducing plant features to model-based
testing of programmable controllers in automation systems,” Control
Engineering Practice, vol. 90, pp. 301–310, 2019.
[15] S. Rösch and B. Vogel-Heuser, “A light-weight fault injection approach
to test automated production system PLC software in industrial practice,”
Control Engineering Practice, vol. 58, pp. 12–23, 2017.
[16] C. Schotten and H. Meyr, “Test point insertion for an area efficient
BIST,” in IEEE International Test Conference (TC), 1995, pp. 515–523.
[17] Y. J. Huang, J. F. Li, J. J. Chen, D. M. Kwai, Y. F. Chou, and C. W.
Wu, “A built-in self-test scheme for the post-bond test of TSVs in 3D
ICs,” in 29th IEEE VLSI Test Symp., 2011, pp. 20–25.
[18] A. Koneru, S. Kannan, and K. Chakrabarty, “A Design-for-Test Solution
Based on Dedicated Test Layers and Test Scheduling for Monolithic 3D
Integrated Circuits,” IEEE Transactions on Computer-Aided Design of
Integrated Circuits and Systems, pp. 1942–1955, 2018.
[19] G. Xie, G. Zeng, Y. Liu, J. Zhou, R. Li, and K. Li, “Fast Functional
Safety Verification for Distributed Automotive Applications during Early
Design Phase,” IEEE Transactions on Industrial Electronics, vol. 65,
no. 5, pp. 4378–4391, 2018.
[20] S. Fischmeister and P. Lam, “Time-aware instrumentation of embedded
software,” IEEE Transactions on Industrial Informatics, vol. 6, no. 4,
pp. 652–663, 2010.
1551-3203 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF BIRMINGHAM. Downloaded on May 11,2020 at 12:05:12 UTC from IEEE Xplore. Restrictions apply.
10
[21] Z. Gu, C. Wang, M. Zhang, and Z. Wu, “WCET-Aware Partial Control
Flow Checking for Soft Error Protection in Resource-Constrained Real-
Time Embedded Systems,” IEEE Transactions on Industrial Electronics,
vol. 61, no. 10, pp. 5652–5661, 2014.
[22] J. Provost, J.-M. Roussel, and J.-M. Faure, “Translating Grafcet speci-
fications into Mealy machines for conformance test purposes,” Control
Engineering Practice, vol. 19, no. 9, pp. 947–957, 2011.
[23] C. G. Cassandras and S. Lafortune, Introduction to discrete event
systems, 2nd ed. Springer Science & Business Media, 2009.
[24] D. Lee and M. Yannakakis, “Principles and methods of testing finite
state machines - a survey,” Proceedings of the IEEE, vol. 84, no. 8, pp.
1090–1123, 1996.
[25] I. K. Voyiatzis and D. J. Kavvadias, “On the generation of SIC pairs
in optimal time,” IEEE Transactions on Computers, vol. 64, no. 10, pp.
2891–2901, 2015.
[26] J. Provost, J.-M. Roussel, and J.-M. Faure, “Generation of single input
change test sequences for conformance test of programmable logic
controllers,” IEEE Transactions on Industrial Informatics, vol. 10, no. 3,
pp. 1696–1704, 2014.
[27] ——, “Technical report on Conformance Test of Programmable Logic
Controllers – Execution of Minimum-Length Test Sequences,” LURPA,
ENS Cachan, France, Cachan, Tech. Rep., 2014.
[28] A. Guignard and J.-M. Faure, “A conformance relation for model-based
testing of PLC,” in 12th Int. Workshop on Discrete Event Systems,
Cachan, may 2014, pp. 412–419.
[29] L. Feng, K. Cai, and W. M. Wonham, “A structural approach to the non-
blocking supervisory control of discrete-event systems,” International
Journal of Advanced Manufacturing Technology, vol. 41, no. 11-12, pp.
1152–1168, 2009.
Canlong Ma received a B.Sc. degree in Mecha-
tronics from Tongji University, China in 2011,
a M.Sc. degree in Automation Engineering from
RWTH Aachen University, Germany in 2013, and
a Ph.D. degree in the Assistant Professorship for
Safe Embedded Systems at Technical University
of Munich, Germany in 2019. The Ph.D. disserta-
tion title is “Advances in Model-Based Testing of
Programmable Controllers: Automatic Test Genera-
tion using Design-to-Test and Plant Features”. He
is currently a functional safety expert at Validas
AG in Munich, Germany. His current research interests include automated
verification and unit- and integration testing of automation and automotive
systems.
Julien Provost received a Ph.D. degree from École
normale supérieure de Cachan, France, in 2011.
He then joined Chalmers University of Technology,
Sweden, as a Post-Doctoral Researcher for two
years. He was an Assistant Professor at Technical
University of Munich, Germany where he held the
Assistant Professorship for Safe Embedded Sys-
tems from 2013 to 2019. Currently he works as
a functional safety expert at Validas AG in Mu-
nich, Germany. His research interest focuses on
formal methods for verification & validation (model-
checking to black-box testing); formalization and control of discrete event
systems (requirements formalization, model-based design, and automatic code
generation); and application to cyber-physical systems, automotive, industry
4.0, smart home for the aging society.