See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/366903699

Coverage Testing of Industrial Simulink Models Using Monte-Carlo and SMT-

Based Methods

Conference Paper · December 2022

DOI: 10.1109/QRS57517.2022.00050

CITATIONS READS
0 75

6 authors, including:

Daisuke Ishii Takashi Tomita

Japan Advanced Institute of Science and Technology Japan Advanced Institute of Science and Technology
34 PUBLICATIONS 115 CITATIONS 28 PUBLICATIONS 100 CITATIONS

SEE PROFILE SEE PROFILE

Toshiaki Aoki Ngoc Thi Bich Do

Japan Advanced Institute of Science and Technology Posts and Telecommunications Institute of Technology
101 PUBLICATIONS 418 CITATIONS 12 PUBLICATIONS 63 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Daisuke Ishii on 06 January 2023.

The user has requested enhancement of the downloaded file.

IEEE Copyright Notice
Copyright (c) 2022 IEEE.
Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future
media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works,
for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

Accepted to be published in the 22nd International Conference on Software Quality, Reliability, and Security (QRS), 2022.
 Coverage Testing of Industrial Simulink Models
Using Monte-Carlo and SMT-Based Methods

Daisuke Ishii1,∗ , Takashi Tomita1 , Toshiaki Aoki1 , The Quyen Ngo2 , Thi Bich Ngoc Do3 , Hideaki Takai4
1
Japan Advanced Institute of Science and Technology, Ishikawa, Japan
2
VNU University of Science, Hanoi, Vietnam
3
Posts and Telecommunications Institute of Technology, Hanoi, Vietnam
4
GAIO Tech. Co., Tokyo, Japan
{dsksh,tomita,toshiaki}@jaist.ac.jp, ngoquyenbg@hus.edu.vn, ngocdtb@ptit.edu.vn, takai.h@gaio.co.jp
*corresponding author

Abstract—Simulink is a popular tool for modeling cyber-
physical systems. As more models are produced in industry,
automated quality assurance of models becomes increasingly
important. This paper describes an empirical evaluation of
four methods for the coverage testing of Simulink models:
A) SimuLink Design Verifier (SLDV), a dedicated official tool;
B) Template-Based Monte-Carlo (TBMC) method, a random test
generation method that utilizes input signal templates; C) SMT-
Based Model Checking (SBMC) method that conducts static
analysis via encoding models into logic formulas; and D) a hybrid
method of B and C. Based on the evaluation results, we carefully
designed the hybrid method to complement the features of TBMC
and SBMC. In the experiments, we have applied the methods to
fourteen models and evaluated their performance. The results
show that the hybrid method achieved better results than SLDV
for several models.
Keywords—MATLAB/Simulink; coverage testing; Monte-Carlo
method; SMT solver
I. I NTRODUCTION
Model-based approach is effective for design and analysis
of systems embedded within cyber-physical systems such as
vehicles and robots [1]. In the development, description of a
CPS model and its verification/validation are important. MAT-
LAB/Simulink [2] is a tool to support such activities; it provides
a graphical modeling language, a numerical simulator, and
other useful toolboxes.
In the model-based development, coverage testing (Sect. III)
is often conducted to certify that a model is simulated ex-
haustively. A MATLAB toolbox SimuLink Design Verifier
(SLDV) [3] is provided for this purpose, which manages a list
of test objectives for a Simulink model and either fulfills an
objective by generating a corresponding test case or detects
that it is a dead logic (DL); a testing process results in
coverage rate, i.e. the ratio of fulfilled and dead items to the
total number of objectives.
In industrial settings, testing of Simulink models often does
not proceed efficiently. First, full coverage of all objectives
may be difficult for industrial models. If an objective depends
on a large model description and/or if test cases that fulfill an
objective are rare, the analysis may become time-consuming.
Also, complex models such as nonlinear digital filters may
result in objectives that are difficult to analyze. If an objective
remains undecided, it will be unknown to testers whether it
is satisfiable or dead. Therefore, testing methods to efficiently
achieve high coverage have been studied extensively, e.g. [4]–
[8]. Second, using a black box tool like SLDV makes it
difficult to understand the process. Testers are often forced to
investigate the cause of long testing time and error messages.
It would be useful if the tool could provide clues.
An effective approach for coverage testing is Monte-
Carlo. We have proposed the Template-Based Monte Carlo
(TBMC) [9] method and it has been shown efficient against
large and complex Simulink models. On the other hand, we
have applied SMT solvers to the Simulink test generation [10],
which is regarded as an invariance verification problem and
verified by an SMT-Based Model Checking (SBMC) method.
To analyze a model and an objective, it encodes them into a
logic formula in two ways: approximately in the domain of
mathematical reals and integers; or, exactly in the domain of
bit vectors and floating-point (FP) numbers.
The objective of this paper is twofold: the first is to clarify
the performance of the coverage testing methods on industrial
Simulink models; the second is to propose a hybrid method
that combines the effectiveness of the TBMC and SBMC
methods. The main contributions are summarized as follows:
• Empirical evaluation results of the four methods (Sect. VI
and VII), which are:
A) SLDV (Sect. IV-A);
B) TBMC method (Sect. IV-B);
C) SBMC method (Sect. IV-C);
D) hybrid TBMC and SBMC method.
We have experimented to apply the methods to fourteen
Simulink models that consist of scalable models and
industrial models. Based on the experimental results and
the properties of the models, we evaluate the performance
and the characteristics of each method and make compar-
isons among methods.
• A hybrid TBMC and SBMC method (Sect. V). We explain
the design and implementation of the method. Then, we
discuss the correctness, efficiency and advantages of the
method over other methods in the context of experimental
results.
 The paper is organized as follows. Sect. II introduces
1
the basics of Simulink. Sect. III explains coverage testing. >= 2 1
In1 C0
Sect. IV introduces the existing methods and Sect. V describes 1 1
the hybrid method. The experimental results are reported in Sw
2 2 Out1
Sect. VI and discussed in Sect. VII. Sections IV,VI and VII In2
Add
UD ex2
contain a subsection for each method. Sect. VIII describes
threats to validity. Sect. IX reviews the related studies. (a) Model ex1: counter with subsystem ex2.

II. S IMULINK
1
Simulink [2] is a MATLAB toolbox based on a graphical In1 <= 0
Pos 0
language for describing models of synchronous and hybrid Leq0
1
systems. Simulink models are timed and simulated on a C0 1
Out1
timeline discretized with sample timings. In this work, we >0
Or
assume a setting where sample time is fixed in a simulation. 2
Sine
*, 2 Gt0
By considering the sample time as one step, models can be In2 MPSw
regarded as a state transition system (Sect. II-B). Simulink
(b) Model ex2: system with a multi-port switch and conditions.
diagrams are structured as hierarchical directed graphs with
edges, called lines, and nodes, called blocks. 1.0
The numerical simulator of Simulink computes the output 0.0 Sw
signals, given a model and a bounded time horizon. In this 1.0
paper, we regard a signal as a bounded sequence of output 0.0 MPSw
values; j-th value is output at time j × st (j ≥ 0 and st is a
true Out1
configured sample time). false
0 1 2 3 4 5
A. Example Simulink Models (c) Output signals when a constant signal of value 1 is fed to In1 of ex1.
Fig. 1 exemplifies Simulink models. Model ex1 (Fig. 1a) Sample time is configured as st = 1.
describes a counter connected to a subsystem. The counter is Figure 1. Example Simulink models.
described by a feedback loop that consists of block instances of
types Inport, Contant, Unit Delay, Add and Switch
(they are named In1, C0, UD, Add and Sw, respectively); it denote their domain D1 × · · · × Dn by D(v). A transition
represents step-wise addition of input values and reset to 0 system (I, T ) consists of an initial condition I ⊆ D(s) and
when the sum becomes 2. a transition relation T ⊆ D(s)×D(s)×D(i)×D(o).
The Subsystem block named ex2 is defined by another Signals obtained by numerical simulation are formalized by
Simulink diagram shown in Fig. 1b. Model ex2 contains execution paths of transition systems.
a Saturate block (Pos) that carries negative values to 0
and a Multiport Switch block (MPSw) that relays one Definition 2. Let σj , ιj and oj be values in D(s), D(i)
of the three inputs based on the first input signal. It also and D(o), respectively (j ∈ {−1, . . . , k −1}). Given (I, T ),
contains logical blocks (Compare To Zero1 and Logic) execution paths (or executions) of length k are
in the downstream that output Boolean signals. ι0 /o0 ι1 /o1 ιk−1 /ok−1
Each block is configured with its parameters such as the σ−1 −−−→ σ0 −−−→ σ1 · · · σk−2 −−−−−−−→ σk−1 ,
value 0 output by C0 and the branching condition i2 ≥ 2 of
Sw (i2 represents the second input signal). where I(σ−1 ) holds and T (σj−1 , σj , ιj , oj ) holds for every
Example signals flowing through the model ex1 are shown j ∈ {0, . . . , k−1}. Input, output and state signals are the traces
in Fig. 1c. Three plots show the output signals of i) block ι0 · · · ιk−1 , o0 · · · ok−1 , and σ−1 · · · σk−1 of an execution path.
Sw of ex1, ii) block MPSw of ex2, and iii) subsystem ex2 The input and output signal values at the initial time are
(or the entire system). The lines depict piecewise continuous represented by ι0 and o0 .
trajectories when the model is interpreted as a hybrid system.
C. Formalization Example
B. Formalization of Simulink Models
As the semantics of Simulink, we consider translation from Each block instance in a model is regarded as a transition
Simulink models to transition systems, as in [10]. It regards a system. For example, block Add in Fig. 1a is translated as
Simulink model as a transition system. (IAdd , TAdd ), where:

Definition 1. Assume input, output, and state variables, i, o IAdd (s) :⇔ ⊤, TAdd (s, s′ , i, o) :⇔ o = i1 + i2 .
and s. For a (vector-valued) variable v = (v1 , . . . , vn ), we
Note that i = (i1 , i2 ) and state variables are not used in the
1 Provided as a built-in subsystem that contains a Compare block. rhs since the block is stateless. ⊤ represents “no constraint.”
Block UD, which is a stateful block, is translated as:
IUD (s) :⇔ s = 0, TUD (s, s′ , i, o) :⇔ s′ = i ∧ o = s.
A user-defined model such as ex1 is translated as a system
(Iex1 , Tex1 ) that combines subordinate transition systems:
Iex1 (s) :⇔ IUD (s1 ) ∧ Iex2 (s2 ),
Tex1 (s, s′ , i, o) :⇔ ∃oC0 , ∃oUD , ∃oAdd , ∃oSw ,
TC0 ((), (), (), oC0 ) ∧ TUD (s1 , s′1 , oSw , oUD )
∧ TAdd ((), (), (i1 , oUD ), oAdd ) ∧ Tex2 (s2 , s′2 , (oSw , i2 ), o)
∧ TSw ((), (), (oC0 , oAdd , oAdd ), oSw ).
Note that the state variables s/s′ combine all the state variables
of subsystems (s1 /s′1 of UD and s2 /s′2 of S2). Value () is used
to represent an “empty” state. The formalization in [10] is
slightly different; the rhs of In and Tn for blocks (e.g. n = UD
and n = C0) is expanded in a user-defined system description
(e.g. the rhs of Iex1 and Tex1 ).
An execution path of ex1 can be obtained according to
Def. 2, given an initial state σ−1 = (0, 0) and an input signal
11 · · · for example. Signals can be depicted as the points in
Fig 1c. For example, the first plot corresponds to the values
of oSw in the definition of Tex1 .
III. C OVERAGE T ESTING AND D EAD -L OGIC D ETECTION
This section overviews the notions and process of the
Simulink coverage testing.
A. Coverage Testing of Simulink Models
Once a system is modeled, developers should simulate the
executions to verify their correctness and performance, and to
validate the model. Coverage testing is important to ensure that
the executions of the model have been exhaustively simulated.
The goal of coverage testing is to achieve higher coverage,
preferably full coverage, by preparing an appropriate test suite
(i.e. a set of input signals). The Simulink toolchain offers
tool support for this purpose. SLDV (Sect. IV-A) provides test
generation and various analysis functionalities based on formal
analysis of models. Simulink Coverage (SLC)2 is another tool
for the analysis of coverage achieved by a set of test cases.
The tools assume coverage criteria that are intended to
measure the ratio of simulated behavior to total behavior.
For Simulink, DC (decision coverage), CC (condition —) and
MC/DC (modified condition and decision —) are the criteria
often used. Intuitively, DC and CC require the coverage of
input and output domains (typed as Boolean in most cases)
of a target block; MC/DC requires to observe that each input
can affect the output by changing its value.
To confirm the coverage, behavioral variations of models
are quantified by counting how many objectives are fulfilled.
Test objectives are specified for block instances based on their
block types3 and they are formalized as (time-independent)
predicates on the target block.
2 https://www.mathworks.com/products/simulink-coverage.html.
3 https://www.mathworks.com/help/slcoverage/ug/
model-objects-that-receive-coverage.html.
Definition 3. A test objective associated with a Simulink block
(I, T ) is a predicate in D(s) × D(s) × D(i) × D(o).
An objective is fulfilled (feasible, or satisfied) iff it holds
at least at a step of an execution. In the coverage testing, the
coverage rate (Def. 4), the ratio of fulfilled versus total number
of objectives, is used as a metric. The 100% coverage rate is
called full coverage.
B. Example Test Objectives
SLDV detects the following 21 objectives in the model ex1:
• Sw: 2 DC objectives.
• ex2/Pos: 4 DC objectives.
• ex2/MPSw: 3 DC objectives.
• ex2/Leq0/Compare: 2 CC objectives.
• ex2/Gt0/Compare: 2 CC objectives.
• ex2/Or: 4 CC and 4 MC/DC objectives.
The DC criterion requires the Switch blocks to observe
values for both true and false cases fed to its control inport,
at least once in an execution. For Sw, they are represented by
two objectives i2 ≥ 2 and ¬(i2 ≥ 2), where i2 is the second
element of the input variable of (ISw , TSw ).
Logic blocks have CC and MC/DC objectives. Each CC
objective corresponds to the true or false case of each input; for
ex2/Or, they are ij and ¬ij , where j ∈ {1, 2}. Additionally,
2 MC/DC objectives are considered for each inport; for the
first inport, they are the true case i1 ∧ ¬i2 and the false case
¬i1 ∧ ¬i2 .
Unlike SLDV, SLC and our method detect 19 objectives in
ex1. For MC/DC objectives, they count only when both true
and false cases are observed.
C. Dead-Logic Detection (DLD)
DLD is a complementary task to coverage test generation.
It aims to detect infeasible objectives that cannot be satisfied
by any test cases in a model under testing. For each obj in the
set of objectives, it can be considered as a safety verification
task described by the LTL model checking problem (I, T ) |=
□¬obj , where (I, T ) is a model containing the target block.
Namely, it examines an arbitrary execution path of (I, T ) and
checks that obj does not hold at any step. More precisely,
execution paths of arbitrary length k are considered and it
is checked whether obj (sj−1 , sj , ij , oj ) is satisfiable, where
0 ≤ j ≤ k−1.
Importantly, DLs of a Simulink model can be detected
locally. The method in [10] conducts the model checking
(I, T ) |= □¬obj on a subsystem (I, T ), and gradually
expands the target to surrounding subsystems until the verifi-
cation succeeds. For example, an MC/DC objective to check
that both inputs of ex2/Or are false can be verified as DL
by only analyzing the subsystem ex2 (see also the discussion
in Sect. VII-D). A criterion to check that the first input of
ex2/MPSw becomes 2 is satisfiable if the target system is
ex2, but it is DL if ex1 is the system under analysis since
block Sw always outputs values less than 2. On the other
hand, to prove that obj is feasible, it is required to witness
an execution path of the top-level system.
 Finally, in this paper, we define the coverage rate as follows:
Definition 4. The coverage rate is ((# L) + (# D))/(# O)
where # L, # D and # O denote the number of fulfilled, dead
and total objectives.
IV. E XISTING M ETHODS FOR C OVERAGE T ESTING
This section describes the following three methods/tools
for Simulink coverage testing: A) SLDV; B) TBMC method;
C) SBMC method.
A. SimuLink Design Verifier (SLDV)
SLDV [3] is a practical black box tool for analyzing
Simulink models. SLDV provides three functionalities ex-
plained below, each of which is performed automatically after
a few settings. The process of two of the three is shown in
Fig. 2. A generated report describes the success or failure of
each objective. SLDV is based on “formal methods,” and also
seems to apply approximations in the analysis.
First, “test generation (TG)” produces test cases to satisfy
(fulfill) coverage criteria; at the same time, it performs DLD
and measures the achieved coverage rate. Along with the
generated test cases, the method outputs a report that contains
the numbers of satisfied, unsatisfiable and undecided objec-
tives. During a process, it may report that some objectives
are “unsatisfiable under approximation,” which implies the
analysis with approximation.
Second, “design error detection (DED)” provides a DLD
function “without requiring extensive tests or simulation runs.”
DED runs efficiently but it may exit without detecting DLs that
exist. In the experiment, we performed DLD by disabling the
detection for other design errors; option “identify active logic”
was also disabled because it seemed to work the same as the
DLD conducted by TG when enabled. The method outputs the
number of detected DLs.
Third, “property proving (PP)” is provided for verification
of properties on signals instrumented in models. We did not
use this function in our experiment.
Begin Begin
Model,
Model
Sim. time
Design
Test
error
generation
detection
Coverage info,
List of DLs
Test suite
End End
(a) Test generation (TG). (b) Design error detection (DED).
Figure 2. The process of SLDV.
B. Template-Based Monte-Carlo (TBMC) Method
The TBMC method [9], [11], [12] has been proposed for
coverage testing of Simulink models. A difficulty of the
Monte-Carlo method in this domain lies in its handling of the
vast input signal space efficiently. To this end, TBMC assumes
signal templates to limit the number of candidate input signals.
It uses eight signal templates. Each input is represented
independently as a scalar-valued signal parameterized by a
set of time-invariant variables. In the experiments reported in
[9], [11], [12], TBMC achieved higher coverage than SLDV
for several industrial models within a time limit.
The TBMC process is illustrated in Fig. 3a. The inputs
for TBMC are a model, a termination condition, such as a
condition for the number of trials, and a simulation time hori-
zon. The following process is repeated until the termination
condition is met.
1) Constant propagation in the model description as a
preprocess. It collects the constant values in the model
and reflects them to the domain of input signal values.
The domain of each port is represented as a set of
intervals.
2) Generation of input signals by a random and inde-
pendent selection of a template for each dimension of
each inport.4 Then each parameter of the templates are
instantiated with a random value.
3) Simulation using Simulink for the specified time length.
4) Computation of an (accumulated) coverage rate based
on the simulation result and the objectives in the model.
In Step 2, a template is randomly assigned to each element
of each inport of the model. We use the following eight
templates that are defined in Table 1 in [9].
• Constant(v): constant signal of value v.
• Linear(b, a): constant-rate signal among values b to a.
• NLinear(n, v1 , . . . , vn+2 , t1 , . . . , tn ): piece-wise linear
signal with n switching points.
• Step(b, a, t): single-step signal whose value switches
from b to a at time t.
• NStep(n, v1 , . . . , vn+1 , t1 , . . . , tn ): n-step signal.
• Sine(a, b, f, p): sine signal with the range [b, a], fre-
quency f and phase shift p.
• Square(a, b, f, p, d): square signal. d is the duty cycle.
• Triangle(a, b, f, p, d): triangle signal.
In the experiment, we assume the input signal values are within
[−104 , 104 ], n ∈ [1, 10], and f ≤ 1024(Hz).
Tool implementation. PROMPT [13] is a proprietary tool for
testing Simulink models based on TBMC. It is realized as a set
of MATLAB scripts that includes a static analyzer of model
descriptions, a mechanism to instrument loggers in models, a
signal builder extension with templates, etc. During repetitive
simulations, measuring an achieved coverage efficiently is
essential for TBMC. Therefore, PROMPT uses a dedicated
coverage measurer instead of using SLDV or SLC. See [9],
[12] for the implementation details.
C. SMT-Based Model Checking (SBMC) Method
SMT (satisfiability modulo theories) solvers are automated
provers for the satisfiability of logic formulas that involve
4 Optionally, possible templates can be configured for each inport but such
configurations were not made in our experiments.
 Begin Begin
Model, Model,
Begin
Term. cond., Sim. time Encoding mode,
Objective obj Model,
Preprocess
Encode the model and obj Sim. time
For each i in [1, ) approximately/exactly
TBMC
Template selection For some (k, d) in [1, ) [1, dmax]
and instantiation Obj. list
Conduct Modified
Simulation k-induction for Conclusive SBMC
the level d No
?
Coverage info,
Coverage Yes
Terminate (k, d) Test suite
measurement
No ? End
i
Yes Coverage info, Coverage info,
Test suite Test suite
End End

(a) TBMC method. (b) SBMC method. (c) Hybrid method.

Figure 3. The process of the TBMC, SBMC and hybrid methods.

predicates in various theories e.g. integer, real, bit-vector and 1 ;; Initial condition.
FP number arithmetic. Model checking is a representative 2 (define-fun init_ex1 ((s Real)) Bool
application of SMT solvers [14]. In this paper, we use Z3 [15], 3 (= s 0) )
4 ;; Transition relation.
one of the state-of-the-art SMT solvers.
5 (define-fun trans_ex1 ( (s@0 Real) (s@1 Real)
The SBMC method [10] has been proposed as a verification 6 (i1 Real) (i2 Real) (o Real) ) Bool
tool using an SMT solver for the objectives in Simulink 7 (exists ((o_add Real) (o_sw Real))
models. The process of SBMC (Fig. 3b) mainly consists of 8 (and (= s@1 o_sw)
the following two steps. The inputs are a model, an encoding 9 (= o_add (+ 1 s@0))
10 (= o_sw (ite (>= o_add 2) 0 o_add))
mode that is either “approximate” or “exact,” and an objective.
11 (trans_ex2 o_sw i2 o) )))
1) Encode the target model and the objective into SMT-
LIB,5 an input format of SMT solvers. Encoded results
Figure 4. Example of encoding Tex1 in SMT-LIB.
describe the definition of the transition systems (I, T )
for the target model and its subsystems. The definitions
in Sect. II-B can be directly described using the function constructs an intermediate data structure. The structure is
definition notation in SMT-LIB (Fig. 4). Also, a monitor simplified by slicing the upstream of the target block. Its back-
for the objective is instrumented in the definition of T end is an SMT-LIB printer for the definition of (I, T ) and
so that whether the objective holds at each step can be for execution paths. The approximate encoder maps Simulink
checked. integer types to the SMT-LIB sort Int and FP number types
2) Conduct an SMT-based test generation by verifying the to sort Real. The exact encoder maps each integer type to a
invariance of ¬obj ; its success indicates that obj is corresponding bit-vector sort and each FP number type to a
DL; its failure results in a counterexample that can be corresponding sort of the FloatingPoint logic.
utilized as a test case. Model checking is based on the The tool supports the encoding of the following 38 block
k-induction method [16] and it is conducted subsystem- types (divided by category):
wise for efficiency. The loop in Fig. 3b enumerates a
• Ports & subsystems: Inport, Outport, Subsystem,
number d of subsystem hierarchies and a bound k of the
ActionPort, EnablePort, TriggerPort.
execution path lengths to be analyzed. Then, execution
• Sources: Constant.
paths of length k or less of the subsystem d levels above
• Signal attributes: RateTransition,
the target block are encoded and analyzed using the
DataTypeConversion, SignalConversion.
SMT solver.
• Signal routing: Mux, Demux, Concatenate, Switch,
Tool implementation. SBMC has been implemented in MAT- MultiPortSwitch, Goto, From, Selector, If,
LAB as an add-on to PROMPT [10]. Implementation consists Merge, BusCreator, BusSelector.
of encoding scripts and a script that implements the process • Discrete: Delay, UnitDelay, Memory.
of Fig. 3b. • Logic: RelationalOperator, Compare To
The encoder analyzes the graph structure of a model Constant, Compare To Zero, Logic.
description and the structure of subsystem hierarchy and • Discontinuities: HitCross, Saturate, Wrap To
Zero.
5 http://smtlib.cs.uiowa.edu. • Math operations: Abs, Sum, Product, Gain, MinMax.
 • Continuous: DiscreteIntegrator.
Note that not all parameter settings have been supported. The
encoder checks whether each block instance in a model is of
a supported type. If a block is not supported, it is stubbed, i.e.
represented by an unconstrained variable.
The verification script implements the loop in a breadth-
first fashion. It gradually increments both a bound k for
unrolled path lengths and the depth d of subsystem levels. The
verification process communicates with an external server that
wraps Z3 via a TCP socket.
V. H YBRID TBMC AND SBMC M ETHOD
In this paper, we propose a hybrid method that conducts
the TBMC and SBMC methods in sequence as illustrated
in Fig. 3c. This process is designed based on the following
findings from the experiments described in Sect. VI: 1) for
most of the objectives, test generation with TBMC is more
efficient than SBMC; 2) SBMC is able to detect DL and also
may find test cases that are difficult to find with TBMC;
3) SBMC is time consuming and a process for a single
objective may run out of time; 4) SBMC with approximate
encoding is more efficient in terms of execution time than
with exact encoding.
How to switch the two methods is described in Sect. V-A.
TBMC and SBMC used in the hybrid method basically follow
the methods proposed in [9]–[12]. In addition, TBMC includes
constant propagation as a preprocess; SBMC is modified as
explained in Sect. V-B. Additional implementation efforts are
described in Sect. V-C.
A. Condition for Switching between TBMC and SBMC
For many target models, after TBMC fulfills the objectives
it can handle, the coverage rate does not increase with the
number of random test generations. We expect that it saturates
exponentially as shown experimentally in Sect. VI-B and
discussed in Sect. VII-B. Therefore, we check a logarithmic
condition on the number of failures (# F), which counts
random simulations that do not contribute to the coverage,
and then switch to the SBMC method. To this end, TBMC is
set with the termination criterion (# F) ≥ c1 − c2 log t, where
t represents the elapsed time of the process. In the experiment
in Sect. VI, the coefficients were set as c1 = 20 and c2 = 2.5;
see the discussions in Sect. VII-B and VII-D.
B. Iterative Deepening SBMC
In the first phase of Fig. 3c, a TBMC process will result
in a list of uncovered objectives. The hybrid method uses a
modified SBMC method in Fig. 5 that can efficiently handle
a list of objectives, aiming at fulfilling as many objectives as
possible within a time limit. Also, regarding the finding 4, we
conduct the approximate analysis first and conduct the exact
analysis second, only to prove DLs.
The modified SBMC process in Fig. 5 is given a list of
objectives and manages the list by associating an analysis
status to each objective. The status is one of the following:
Begin
Model, Objective list L
For some (k, d) in [1, ∞) × [1, dmax]
For each inconclusive obj in L
Encode approximately and
verify obj with k and d
obj
For each possibly dead obj in L
Encode exactly and
verify obj with k and d All
concluded
obj ?
No
Yes
(k, d)
Coverage info, Test suite
End
Figure 5. The process of the modified SBMC.
a) inconclusive; b) a test case is generated; c) DL detected ap-
proximately; d) DL proved (with exact encoding). The process
is implemented as triple loops that iteratively enumerate the list
elements while gradually increasing the values of parameters
k and d (the first and second loops are combined in Fig. 5).
1) The outmost loop manipulates the bound k, starting
with value 1 and doubling its value at every iteration.
Therefore, paths of length 2j−1 to 2j is verified in the
j-th iteration.
2) The second loop manipulates the depth d by increment-
ing its value from 0 (parent subsystem of the target
block) to dmax (top-level system). It only considers the
objectives of status c that are possiblly DL.
3) The third and innermost two loops enumerate the ob-
jectives and invoke SBMC in either the approximate or
exact mode using the current values of k and d.
The loops skip the objectives with a conclusive status and
terminate when all are conclusive.
C. Tool Implementation
The hybrid method has been implemented in MATLAB as
an add-on to PROMPT.
Our implementation adds to TBMC a function to generate
a list of objectives in a model. For example, a DC and a CC
objectives in ex1 can be output as the following MATLAB
cell arrays:
{ {’ex1/Sw’,’DC’,1,1}, ’vi{2} >= 2’ }
{ {’ex1/ex2/Gt0/Compare’,’CC’,1,0},
’˜logical(vo{1})’ }
They consist of the target block path, criterion kind, element
index of the signal, true (1) or false (0) case, and the objective
represented as a MATLAB condition.
In SBMC, we have additionally implemented a simple
conformance checking function that checks whether a target
objective can be fulfilled in Simulink when a counterexample
found with Z3 is simulated. It first transfers the input signal
of a counterexample to a SignalBuilder block connected TABLE I
to the input of the target model and invokes a simulation. S IZE AND COMPLEXITY OF THE MODELS
Then, it obtains log data from the model and checks whether “s’s” and “h’s” are short for “subsystems” and “hierarchies.”
the objective (the condition in the list) is fulfilled by the Model # blocks # s’s # h’s # DC # CC # MC/DC
corresponding signal.
ex1 26 3 3 9 8 4
si1 12 1 2 4 2 0
VI. E XPERIMENTAL R ESULTS si2 20 2 2 8 2 0
si3 28 3 2 12 2 0
This section describes empirical evaluation results of the nc8 74 7 8 32 6 2
coverage testing methods. The main purpose of the experi- nc16 146 15 16 64 6 4
ments is to benchmark the four methods and in particular to nc32 290 31 32 128 6 4
compare the performance of SLDV and the hybrid TBMC lm0 479 39 5 42 186 86
lm1 279 29 5 48 32 20
and SBMC method. Comparison of the tools also validates lm3 289 18 5 24 4 0
the correctness of their results. The statistics are shown in lm5 709 187 6 128 0 0
Tables II and III, and the achieved coverage rates are also s0 598 33 6 159 108 48
shown in Fig. 7 for comparison. A detailed discussion follows s1 380 18 4 137 154 54
in Sect. VII. s2 310 28 7 84 100 64
Experiments were conducted on 64-bit Ubuntu 20.04 virtual
machines (with 4 cores and 16GB RAM), running on a 2.2GHz
Intel Xeon E5-2650v4 processor (12 cores) with 128GB RAM.
Each tool was not supposed to use parallel processing. We RelationalOperator to check that the final output
used MATLAB R2022a and Z3 4.8.14. Execution time for is less than 1; its violation should be DL.
each process was limited to 1 hour. • ncn : nested counter. The intension of this model is that
Evaluated features. In the experiment, we mainly evaluated the entire system hierarchy must be analyzed for DLD.
the performance of the target methods on coverage testing, The model describes n counters, each with a holding
represented by achieved coverage rates and time (for full- mode and resets the value after 3 increments. They are
coverage runs). In the discussion, we compare the differences nested in such a way that each level increments when
between the results of each tool in terms of correctness. We the inner counter resets; the i-th level from the inside
also consider the characteristics of handled models (Table I). increments approximately every 3i steps. When the top-
level counter value reaches 2, it will force a hold on the
The comprehensibility of the process of each tool was briefly
innermost counter to prevent a reset; hence, the top-level
evaluated. As an indicator of the process, the proposed method
reset is DL.
outputs the quantities “# of trials” (Fig. 6) and kmax (Table III),
• lm0/1/3/5: models from the Lockheed Martin challenge
while SLDV does not provide such an indicator. We also
problems [17], [18]. We selected models containing DLs;
evaluated how incomplete analyses for some objectives were
also, we excluded models using externally-provided block
reported; SLDV output warning messages; DLD with SBMC
sets and matrix division. Some models are configured
was incomplete if it did not succeed with the exact encoding.
with continuous timeline but we modified them to assume
Apart from the effort required for basic settings (e.g. for
a fixed-step time. They consist of a triplex signal monitor
switching condition and simulation time), the tools were
(lm0), a finite state machine (lm1), a feedback control
comparable in usability, as each tool automatically performed
loop regulators (lm3), and a two-input single-output
coverage testing on a given model.
predictor neural network with two hidden layers (lm5).
We did not evaluate other features such as the user interface,
All but lm1 deal with matrix signals.
the format of results, the functionality to link with other
• s0/1/2: sample models taken from industrial sites.
modeling functions, and the maturity of the tools. A more
These are confidential models used to evaluate PROMPT.
versatile comparison of methods is a future issue.
s0 models a classifier for a driving input. It contains
Target models. We consider fourteen instances of Simulink Lookup Table and Quantizer blocks that make
models, which are summarized as follows; the first three items static analysis difficult. s1 models a composite digital
introduce artificial models and the last two items describe filter. It involves a detector for the synchronization of
practical models. Table I shows the size and complexity of two large counters that happens rarely. s2 computes an
model instances; the numbers of SLDV objectives (# DC, integer signal from two Boolean input signals. It consists
# CC, # MC/DC) are shown as a metric of the complexity. of two units, each modeled as a large feedback loop.
• Model ex1 in Sect. II-A.
• sim : sequential integrators. Models connect m integra- Models sim and ncn are provided by parameterized model
tor circuits in series. We model each circuit to have the schemes; we set the parameters as m ∈ {1, 2, 3} and n ∈
same output domain [0, 1), but to make the degree of {8, 16, 32}. Scripts for generating the instances are available
integration smaller for the backward circuits. We put a at https://github.com/dsksh/sl-examples.
 TABLE II 100
lm0
E XPERIMENTAL RESULT WITH SLDV s0
80

Coverage
The abbreviations “O,” “D,” “L,” “Cov” and “TO” are short for lm1
“objectives,” “dead objectives (logic),” “live (fulfilled) objectives,” 60 lm5
“Coverage rate” and “Time out,” respectively. s1
40 s2
SLDV (DED) SLDV (TG) lm3
Model #O #D Time #L #D Cov. Time 20
100 101 102 103
ex1 21 2 8s 16 5 100% 13s # of trials
si1 6 0 9s 5 1* 83.3% TO
si2 10 0 9s 7 3* 70.0% TO lm3
si3 14 0 9s 9 5* 64.3% TO s2
15 lm1

# of failures
nc8 40 0 10s 33 0 82.5% TO lm5
nc16 74 0 13s 50 1 68.9% TO 10 lm0
s1
nc32 138 0 21s 82 1 60.1% TO ek
lm0 314 1 26s 312 2 100% 18s 5 Switching
lm1 100 3 8s 93 7 100% 10s
condition
lm3 28 8 16s 18 8 92.9% TO 0
lm5 128 6 36s 118 6 96.9% 195s 10 50 100 500 1000
s0 315 15 34s 259 0 82.2*% TO Time
s1 345 0 18s 313 0 90.7% TO
s2 248 12 56s 224 12+4* 95.2% TO Figure 6. Coverage rates achieved by TBMC.

A. SLDV of DLs detected using the exact encoding. Exceptionally for

The result of coverage testing of the target models using
SLDV is summarized in Table II. The column “# O” shows
the number of overall objectives calculated by SLDV. The
section “SLDV (DED)” shows the number of DLs detected
by DED and time required. The section “SLDV (TG)” shows
the result of TG: the numbers of fulfilled objectives and DLs,
the achieved coverage rates and the timing data. DLs of sim
were found “unsatisfiable under approximation” by TG; it
continued to prove DL but the process did not terminate. Some
objectives of s0 were analyzed differently than the SBMC and
hybrid methods (therefore the coverage rates were computed
differently); see Sect. VII-E for details.
B. TBMC Method
TBMC was configured as follows: 1) each random simula-
tion computed an execution path of length 104 × st, where
st is the least sample time set for the model; 2) the process
was terminated when the full coverage was achieved or the
time limit was exceeded. The result is shown in Table III. The
second column shows the number of objectives calculated by
the PROMPT tool. The numbers were slightly different from
SLDV as explained in Sect. III-B. The section “TBMC” shows
the number of objectives and the coverage rates achieved by
TBMC. Fig. 6 illustrates how the coverage rates saturate for
the large models.
C. SBMC Method
For the SBMC method, we prepared the list of all objectives
for each model and applied the method. The time limit was
applied to the model checking process for a single objective.
The section “SBMC” of Table III shows the result of the
SBMC method. The column “# L” shows the number of
objectives satisfied by the SMT solver and then verified by
a Simulink simulation. The column “# D” shows the number
lm5, s0 and s2, the notation x+y* means x and y objectives
were found DL using the exact and approximate encoding,
respectively.
D. Hybrid Method
Finally, we conducted the coverage testing using the hybrid
method. The result is shown in the section “Hybrid” of
Table III. The data are basically the same as those of the
SBMC method; the column “# L” counts only when the
verification with Simulink has succeeded and the notation
x+y* in the column “# D” means the same.
VII. D ISCUSSIONS
This section aims to answer the following questions:
RQ1) what were the advantages of each method (compared
to the others) in the experimental result? RQ2) what were the
disadvantages? RQ3) what were the differences in the results
between the hybrid method and SLDV? RQ1 and RQ2 are
answered per method. RQ3 is answered in Sect. VII-E.
A. SLDV
Answer to RQ1) SLDV (TG) achieved the highest coverage
rates for 8 models. The performance was slightly better than
other tools for industrial models.
Answer to RQ2) For the overall models, it was only able
to achieve full coverage for 3 of them. Also, errors with
insufficient explanation occurred.
We discuss the performance of SLDV in Sect. VII-E.
There were two reasons for the impediments to achieving
full coverage. The first was time loss due to inefficiency.
SLDV seems to conduct the static analysis of models based on
formal methods, which often be time consuming. Second, there
were cases resulted in errors. Issues with approximation were
 TABLE III
E XPERIMENTAL RESULTS WITH TBMC AND SBMC METHODS
The abbreviations are the same as Table II. Tmax represents the maximum time taken for verifying an objective.

TBMC SBMC Hybrid

Model #O #L Cov. #L #D Cov. Time Tmax kmax #L #D Cov. Time
ex1 19 15 78.9% 11 4 78.9% 85.2s 6.7s 1 15 4 100% 48.6–49.5s
si1 6 5 83.3% 5 1 100% 202s 185s 1 5 1 100% 186–195s
si2 10 7 70.0% 7 3 100% 1330s 890s 1 7 3 100% 1220–1273s
si3 14 9 64.3% 9 5 100% 2680s 1340s 1 9 5 100% 3020–3180s
nc8 40 39 97.5% 29 1 75.0% TO TO 157 39 1 100% 38.5–43s
nc16 72 44 61.1% 44 1 62.5% TO TO 121 44 1 62.5% TO
nc32 136 86 63.2% 75 1 55.9% TO TO 165 86 1 64.0% TO
lm0 286 284 99.3% 284 2 100% 4910s 24s 1 273–284 2 96.2–100% 126–840s
lm1 82 75 91.5% 75 7 100% 2450s 41s 4 75 7 100% 686–830s
lm3 36 24 66.7% 24 8 88.9% 1080s 183s 9 24 8 88.9% TO
lm5 88 79 89.8% 36 0+6* 40.9% TO TO 1 77–79 0+6* 87.5–89.8% TO
s0 300 288 96.0% 237 2+9* 79.7% TO TO 2 292–293 2+9* 98.0–98.3% TO
s1 320 285 89.1% 210 1 65.9% TO TO 9 274–283 1 85.9–88.8% TO
s2 208 185 88.9% 49 2+13* 24.5% TO TO 177 183–185 2+10* 88.9–89.9% TO

100 SLDV/DED (D)

Hybrid (D/A) SLDV/TG (D/A)
Hybrid (D)
Hybrid (L) SLDV/TG (D)
50 SBMC (D/A)
SBMC (D) SLDV/TG (L)
SBMC (L)

TBMC (L)
0
ex1 si1 si2 si3 nc8 nc16 nc32 lm0 lm1 lm3 lm5 s0 s1 s2

Figure 7. Coverage rates achieved in the experiments. “D/A” means “DL detected with approximation.”

reported for the objectives of sim . Some objectives of lm5
were “undecided due to nonlinearities” (therefore, the process
was terminated without full coverage). Because SLDV is a
black box tool, it was often difficult to understand the causes
of inefficiency and the reasons of DLs.
B. TBMC Method
Answer to RQ1) Although TBMC could not mark the
highest coverage rates, it fulfilled many of the objectives
efficiently, examining numbers of long execution paths.
Answer to RQ2) Unfulfilled objectives remained, including
DLs and objectives with rare solutions.
TBMC was able to repeat the simulation for 104 steps
efficiently for most models. For example, around 2000 and
340 simulations (trials) were conducted for lm3 and lm5,
respectively. For complex models such as s2, simulation
became inefficient; only 31 or 32 trials were possible for s2.
Fig. 6 shows presumably exponential saturation of the
coverage rates with the number of trials; the figure below
shows the accumulation of # F and the condition versus time.
Because it saturated quickly when TBMC was efficient, we
chose to count the number of failed trials (# F) in the switching
condition of the hybrid method; the coefficients c1 and c2 in
Sect. V-A were set to count a sufficient # F for simple models
(e.g. lm3) and several # F also for complex models (e.g. s2).
The main obstacles to full coverage by TBMC are DLs
and rare test cases. For instance in s2, TBMC could not
find a signal that trusifies two Compare To Constant
blocks that represent equality conditions. In the hybrid setting
(Sect. VII-D), some objectives were fulfilled by SBMC instead
of TBMC.
C. SBMC Method
Answer to RQ1) SBMC achieved full coverage for 5 mod-
els. Notably, it was able to analyze a set of execution paths
rigorously to obtain a shortest test case. Exact analysis of
FP computation was possible.
Answer to RQ2) The efficiency of the process was bad.
With approximate encoding, false results may be obtained.
The column “kmax ” of Table III shows that it analyzed much
shorter paths than those examined by TBMC. The generated
test cases were the shortest ones and the rare test cases were
not missed. On the other hand, the scalability was limited.
Compared to the objectives fulfilled with TBMC, few were
fulfilled only with SBMC; only two objectives of s2. Many
objectives were fulfilled more efficiently by TBMC. On the
other hand, SBMC could detect a number of DLs.
Compared to SLDV, SBMC was better for sim , achieving
full coverage, but the process took considerably more time.
SLDV produced approximation errors in the DLD of sim ,
while SBMC was able to analyze them using the exact en-
coding. We also confirmed that some live and dead objectives
in s0 and s1 were only handled by SBMC. Therefore, using
both tools together is considered effective for dealing with a
variety of objectives.
An SBMC process became inefficient when a model was
encoded into a large formula; such models were ncn , lm5,
s0 and s2. Also, it was inefficient when long execution paths
were needed to be encoded for the models ncn and s2. As
reported in [10], the scalability of the analysis with the exact
encoding was worse than with the approximate encoding. In
the experiment, we observed this fact for several objectives.
Since SBMC applies abstraction and approximation in the
analysis, spurious test cases can be obtained and DLs can be
detected falsely. In the experiment, a number of such test cases
were obtained and then denied with the confirmation process
with a Simulink simulation (such cases are not counted in the
column “# L”). Most of the DLs detected in the first phase of
Fig. 5 were also confirmed as DLs in the second phase using
the exact encoding. Exceptionally, several DLs of s0 were
denied in the second phase (see the next section).
D. Hybrid Method
Answer to RQ1) The hybrid method marked the highest
coverage rates for 9 models, 7 of which had full coverage.
It provided the complementary advantages of TBMC and
SBMC. The switching condition and the iterative deepening
process were confirmed to be effective.
Answer to RQ2) For 5 models, the coverage rates were
not in the first place.
In the first phase of the hybrid method, TBMC fulfills a
number of objectives efficiently, and in the second phase,
SBMC is applied to each of the remaining objectives to check
its satisfiability. The experimental result shows that the process
was able to employ the complementary advantages of TBMC
and SBMC. It is evident from the coverage rates achieved by
each method in Table III; the coverage rates by the hybrid
method were higher than those of the single methods. The
results for nc8 , lm0 and lm1 show a significant reduction in
execution time than SBMC.
We consider that the switching condition in Sect. V-A (also
shown in Fig. 6) was appropriate. For most of the models, the
same “# L” as that of the stand-alone TBMC was achieved
in the first phase of the hybrid method; then, the same “# D”
was achieved again in the second phase. In the process for
lm0, lm5, s1 and s2, “# L” achieved in the first phase was
slightly less than that of TBMC. In that case (except for lm5),
some objectives were fulfilled in a complementary manner in
the second phase.
We also consider that the iterative deepening strategy in
Sect. V-B was appropriately adopted. Even though stand-
alone SBMC processes for some objectives would run out of
time, the hybrid method was able to avoid invoking them; it
prioritized checking other objectives, and achieved reasonable
“# L” and “# D” in the second phase.
E. Comparison between the Hybrid Method and SLDV
Answer to RQ3) The hybrid method and SLDV achieved
higher coverage rates on 6 and 5 models, respectively.
Hybrid ran out of time less frequently than SLDV, but took
more time when both were in time.
The hybrid method was capable of analyzing detailed nu-
merical computation. SLDV sometimes produced confusing
analyses due to short-circuiting of models.
When compared with SLDV, the hybrid method outper-
formed on the artificial models and s0. SLDV outperformed
on 4 of the industrial models, but only by a small margin.
Model instances sim are difficult to obtain their transfer
functions, which may have caused the approximation error
in SLDV. SBMC was able to analyze them by encoding in
step-based formulas. Instances ncn were prepared to check
the scalability of the tools; they scale both in the amount
of model description and in the path lengths to be analyzed.
The efficiency of both tools decreased with the instance size
increased. We consider the scalability of the hybrid method
was slightly better as it achieved better coverage for nc8
and nc32 . For models lm0–2, the coverage achieved by both
methods was similar, but SLDV was more time efficient. For
other industrial models (except for s0), we believe that the
optimized method used by SLDV was effective; for example,
the use of dependencies among objectives and short-circuiting
of graph structures.
The model s0 has an objective that relates to a condition
that is almost unsatisfiable but can be satisfied by a signal of
value NaN (a special FP number). The objective is reproduced
in ex2; an MC/DC false case of the block ex2/Or requires
the both inputs to be false; it is satisfiable if the output value
of MPSw is NaN since NaN ≤ 0 and NaN > 0 do not hold.
By default, our hybrid method accepts test cases that include
the value NaN; however, SLDV does not. The above objective
in s0 (and the depending objectives) was detected as DL by
SLDV/DED and remained undecided by SLDV/TG. SBMC
detected it as DL with approximate encoding and then denied
with exact encoding by a counterexample involving NaN.
Considering the possibility of another system being connected
to the inport, we believe that an analysis including the NaN
case would be useful.
Another difference between SLDV and our method is due
to that SLDV “treats logic blocks as if they are short-circuiting
when analyzing for DL.”6 In the analysis of ex1, SLDV
detected the CC false case for the second inport of ex2/Or
as DL despite ex2/Gt0 could output false. When ex2/Gt0
output false, ex2/Leq0 always output true; therefore, the
second inport was ignored by the short-circuiting of ex2/Or.
Our method does not take into account the short-circuiting.
The above objective was fulfilled in the experiment. While the
analysis with short-circuiting is related to the implementation
of Simulink and can make the DLD more efficient, it also
6 https://www.mathworks.com/help/sldv/ug/
Common-causes-for-dead-logic.html.
makes the results difficult to understand. The result of the
proposed method is simple and more comprehensible.
VIII. T HREATS TO VALIDITY
Generalizability. Are the number and size of the collected
Simulink models sufficient? Collecting industrial Simulink
models for research purpose is difficult and most of the papers
use only a few examples [19], [20]. According to the model
characteristics data [19], the average size and complexity of
the models we have collected are comparable to those in other
experiments. Also, to ensure diversity, models lm1–lm5 and
s0–s2 were obtained from different sources. Therefore, we
consider that our model set is sufficient for the experiment
by current standards. Still, experiments with more models (cf.
[20]) will be a future work.
Is the proposed hybrid method superior to other coverage
testing methods/tools? SLDV, which we compared, is used
as the industry standard tool for testing Simulink models,
and we believe its functionality is state-of-the-art. Comparison
with third-party tools such as Reactis [21]7 and other reseach-
level methods, which would require extensions for industrial
application, is a future work.
Replicability/reproducibility. The replicabilty [19] of the
artifacts and the experimental results in this paper is limited;
accessibility to our models is partial; TBMC and SLDV
are proprietary tools and the SBMC and hybrid tools are
not publicly available. However, this paper and the previous
papers [9]–[12] describe the overall process of the hybrid
method. TBMC and SBMC rely on the Simulink simulator
and SMT solvers, respectively, which are widely available.
IX. R ELATED W ORK
A. Testing Methods
There exist testing methods for Simulink (a survey [22]
briefly discuss coverage testing). Here, we classify them into
symbolic methods and hybrid methods combined with random
testing, and other methods.
Symbolic test generation methods [4], [7], [23], [24] trans-
late Simulink models to constraints or formal models and
perform a symbolic analysis. Some of them use SMT solvers
in the analysis [7], [24], which is performed to obtain a test
case as a solution that satisfies constraints. They can be used
to fulfill the coverage test objectives; however, handling our
examples will require extensions such as to analyze additional
block types; also, their scalability is questionable.
Hybrid (or concolic) methods [5], [6], [25], [26] have been
proposed as an extended approach. As our experimental results
have shown, there is a significant length gap between the
execution paths handled by symbolic and random methods.
The existing work does not address this point in detail and
the solution is unclear. Some methods (e.g. [26]) analyze
a model description without unrolling, and we suspect they
cannot obtain test cases in the form of paths. They also
consider only test generation, not DLD. The SmartTestGen
7 https://reactive-systems.com/simulink-testing-validation.html.
tool [27], [28] combines four approaches of test generation;
one of them considers unreachability of test objectives to guide
the coverage strategy. From their empirical evaluation, the
effectiveness of the tool for the industrial models we consider
is not clear.
Matinnejad et al. [8] have proposed a test generation and
test prioritization method based on a meta-heuristic search that
considers the diversity of signals. Although their test objectives
are different, it is future work to experimentally compare with
our methods and to introduce their techniques in ours.
A technique related to TBMC is falsification methods
for STL properties [29]. Among several implementations, S-
TaLiRo [30] provides a functionality for Simulink models
based on stochastic optimization. Although the methods could
be used for test generation, we consider TBMC is more
optimized due to its preprocessing and the use of templates.
B. SMT-Based Analysis
Related to SBMC, there are formal analysis techniques for
Simulink and other related systems. Unlike our hybrid method
with a random method, the following work considers stand-
alone use. The CoCoSim tool [31] provides a safety model
checking functionality for Simulink models using Kind2 [32],
an SMT-based model checker. The basic process in this paper
is similar but we provide an accurate encoding method and
more support for industrial models. As reported in [10],
applicability of CoCoSim to our examples are limited. BTC
EmbeddedTester [33] performs a bounded model checking
(BMC) against C code generated from Simulink. Another
BMC method [34] that analyzes Simulink models directly has
been proposed. Their applicability to our models is unknown
and comparison is a future work.
C. Dead Code Detection
DLD relates to dead code detection (DCD) for general-
purpose programs. It has been developed as an optimization
technique for compilers [35]. While DCD is based on the
reachability and executability of instructions in the code, we
detect DLs based on the satisfiability of test objectives. As for
Simulink, DCD is important in coverage testing [36]. Model
checking [37] and SMT solvers [38] have been applied and
implemented in code coverage tools.
X. C ONCLUSIONS
We have presented the hybrid TBMC and SBMC method
for Simulink coverage testing that is based on a Monte-Carlo
method and the analysis using an SMT solver. We prepared
fourteen Simulink models that consist of benchmark models
and industry-derived models and conducted experiments to
evaluate the four methods. The experimental results clarified
the advantages and disadvantages of the methods. It was shown
that the hybrid method took advantage of the features of
TBMC and SBMC; it quickly fulfills objectives in the first
phase and analyzes the remaining objectives rigorously in the
second phase. Our hybrid method was competitive with the
standard tool SLDV; hybrid achieved full coverage for 7 of
14 models and the coverage rates were higher than those
of SLDV in 6 models. We also learned that the objectives
that the methods can handle are slightly different, so future
collaboration between the two should be considered.
There are several issues to be addressed in the future. First,
we can improve the cooperation between TBMC and SBMC;
for example, signal template selection can be assisted by SMT
solvers and the search process of an SMT solver can be guided
with a simulation result. We can also integrate other methods
(e.g. SLDV and test prioritization [8]) with the TBMC and
SBMC methods in a composite testing framework. Second,
after improving the performance of the method, we plan to
apply it to larger and more complex Simulink models. Third,
we can conduct experiments on testing based on other cover-
age criteria and other objectives such as those for discovering
vulnerabilities in models.
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Num-
bers 18K11240, 18H03220, 22K11969.
R EFERENCES
[1] E. A. Lee and S. A. Seshia, Introduction to Embedded Systems. A
Cyber-Physical Systems Approach, 2nd ed. MIT Press, 2017. [Online].
Available: http://leeseshia.org
[2] The Mathworks Inc., “Simulink,” 2021. [Online]. Available: https:
//www.mathworks.com/products/simulink.html
[3] ——, “Simulink Design Verifier,” 2021. [Online]. Available: https:
//www.mathworks.com/products/simulink-design-verifier.html
[4] A. A. Gadkari, A. Yeolekar, J. Suresh, S. Ramesh, S. Mohalik, and K. C.
Shashidhar, “AutoMOTGen : Automatic Model Oriented Test Generator
for Embedded Control Systems,” in CAV, ser. LNCS 5123, 2008, pp.
204–208.
[5] M. Satpathy, A. Yeolekar, and S. Ramesh, “Randomized Directed
Testing (REDIRECT) for Simulink/Stateflow Models,” in EMSOFT,
2008, pp. 217–226.
[6] A. Kanade, R. Alur, F. Ivančić, S. Ramesh, S. Sankaranarayanan,
and K. C. Shashidhar, “Generating and analyzing symbolic traces of
simulink/stateflow models,” in CAV, ser. LNCS 5643, 2009, pp. 430–
445.
[7] S. Chakrabarti and S. Ramesh, “SymTest: A framework for symbolic
testing of embedded software,” in ISEC, 2016, pp. 48–58.
[8] R. Matinnejad, S. Nejati, L. C. Briand, and T. Bruckmann, “Test
Generation and Test Prioritization for Simulink Models with Dynamic
Behavior,” IEEE Transactions on Software Engineering, vol. 45, no. 9,
pp. 919–944, 2019.
[9] T. Tomita, D. Ishii, T. Murakami, S. Takeuchi, and T. Aoki, “Template-
Based Monte-Carlo Test-Suite Generation for Large and Complex
Simulink Models,” IEICE Transactions on Fundamentals of Electronics,
Communications and Computer Sciences, vol. E103-A, no. 2, pp. 451–
461, 2020.
[10] D. Ishii, T. Tomita, T. Aoki, T. Q. Ngô, T. B. N. Do, and H. Takai, “SMT-
Based Model Checking of Industrial Simulink Models,” in ICFEM, ser.
LNCS 13478, 2022, pp. 156–172.
[11] T. Tomita, D. Ishii, T. Murakami, S. Takeuchi, and T. Aoki, “Template-
Based Monte-Carlo Test Generation for Simulink Models,” in Seventh
Workshop on Design, Modeling and Evaluation of Cyber Physical
Systems (CyPhy), ser. LNCS 11267, 2017, pp. 63–78.
[12] ——, “A Scalable Monte-Carlo Test-Case Generation Tool for Large
and Complex Simulink Models,” in Workshop on Modelling in Software
Engineering (MiSE), 2019.
[13] Gaio Technology Co. Ltd., “PROMPT,” 2021. [Online]. Available:
https://www.en.gaio.co.jp/products/prompt-2/
[14] A. Biere and D. Kröning, “SAT-Based Model Checking,” in Handbook
of Model Checking. Springer, 2018, ch. 10, pp. 277–303.
[15] L. de Moura and N. Bjørner, “Z3: An Efficient SMT Solver,” in
TACAS, ser. LNCS 4963, 2008, pp. 337–340. [Online]. Available:
https://github.com/Z3Prover/z3
View publication stats
[16] M. Sheeran, S. Singh, and G. Stålmarck, “Checking safety properties
using induction and a SAT-solver,” in FMCAD, ser. LNCS 1954, 2000,
pp. 127–144.
[17] C. Elliott, “Cyber-Physical V&V Challenges for the Evaluation of State
of the Art Model Checkers,” in Safe and Secure Systems and Software
Symposium (S5), 2016.
[18] A. Mavridou, H. Bourbouh, D. Giannakopoulou, T. Pressburger, M. He-
jase, P. L. Garoche, and J. Schumann, “The Ten Lockheed Martin Cyber-
Physical Challenges: Formalized, Analyzed, and Explained,” in IEEE
RE, 2020, pp. 300–310.
[19] A. Boll, N. Vieregg, and T. Kehrer, “Replicability of experimental
tool evaluations in model-based software and systems engineering with
MATLAB/Simulink,” Innovations in Systems and Software Engineering,
2022.
[20] S. A. Chowdhury, L. S. Varghese, S. Mohian, T. T. Johnson, and
C. Csallner, “A Curated Corpus of Simulink Models for Model-based
Empirical Studies,” in SEsCPS, 2018, pp. 45–48.
[21] S. Sims and D. C. Duvarney, “Experience Report : The Reactis Valida-
tion Tool,” in ICFP, 2007, pp. 137–139.
[22] Z. Sadri-Moshkenani, J. Bradley, and G. Rothermel, “Survey on test
case generation, selection and prioritization for cyber-physical systems,”
Software Testing, Verification and Reliability, vol. 32, no. 1, p. 42, 2022.
[23] D. Bhatt, G. Madl, D. Oglesby, and K. Schloegel, “Towards scalable
verification of commercial avionics software,” in AIAA Infotech at
Aerospace 2010, 2010, pp. 1–8.
[24] H. Ren, D. Bhatt, and J. Hvozdovic, “Improving an Industrial Test
Generation Tool Using SMT Solver,” in NFM, ser. LNCS 9690, 2016,
pp. 100–106.
[25] C. S. Pasareanu, J. Schumann, P. Mehlitz, M. Lowry, G. Karsai,
H. Nine, and S. Neema, “Model Based Analysis and Test Generation
for Flight Software,” in Third IEEE International Conference on Space
Mission Challenges for Information Technology, 2009, pp. 83–90.
[Online]. Available: http://ieeexplore.ieee.org/document/5226844/
[26] M. Souza, M. Borges, M. D’Amorim, and C. S. Pǎsǎreanu, “CORAL:
Solving complex constraints for symbolic pathfinder,” in NFM, ser.
LNCS 6617, 2011, pp. 359–374.
[27] P. Peranandam, S. Raviram, M. Satpathy, A. Yeolekar, A. Gadkari, and
S. Ramesh, “An integrated test generation tool for enhanced coverage
of Simulink/Stateflow models,” in DATE. IEEE, 2012, pp. 308–311.
[28] S. Raviram, P. Peranandam, M. Satpathy, and S. Ramesh, “SmartTest-
Gen+ : A Test Suite Booster for Enhanced Structural Coverage,” in
ICTAC, ser. LNCS 7521, 2012, pp. 164–167.
[29] A. Donzé and O. Maler, “Robust Satisfaction of Temporal Logic over
Real-Valued Signals,” in FORMATS, ser. LNCS 6246, 2010, pp. 92–106.
[30] Y. Annpureddy, C. Liu, G. Fainekos, and S. Sankaranarayanan, “S-
TaLiRo: A tool for temporal logic falsification for hybrid systems,” in
TACAS, ser. LNCS 6605, 2011, pp. 254–257.
[31] H. Bourbouh, P.-l. Garoche, T. Loquen, E. Noulard, and C. Pagetti,
“CoCoSim, a code generation framework for control/command applica-
tions,” in ERTS, 2020, pp. 1–11.
[32] A. Champion, A. Mebsout, C. Sticksel, and C. Tinelli, “The KIND
2 Model Checker,” in CAV, ser. LNCS 9780, 2016, pp. 510–517.
[Online]. Available: https://kind2-mc.github.io/kind2/
[33] P. Schrammel, D. Kroening, M. Brain, R. Martins, T. Teige, and
T. Bienmüller, “Incremental bounded model checking for embedded
software,” Formal Aspects of Computing, vol. 29, pp. 911–931, 2017.
[34] P. Filipovikj, G. Rodriguez-Navas, and C. Seceleanu, “Bounded in-
variance checking of simulink models,” in Proceedings of the ACM
Symposium on Applied Computing, 2019, pp. 2168–2177.
[35] K. D. Cooper and L. Torczon, Engineering a Compiler, 2nd ed. Morgan
Kaufmann, 2012.
[36] P. A. V. Hall and J. H. R. May, “Software Unit Test Coverage and
Adequacy,” ACM Computing Surveys, vol. 29, no. 4, pp. 366–427, 1997.
[37] P. Godefroid and K. Sen, “Combining model checking and testing,” in
Handbook of Model Checking. Springer, 2018, ch. 19, pp. 613–649.
[38] D. Kroening and O. Strichman, Decision Procedures, 2nd ed. Springer,
2016.