Professional Documents
Culture Documents
1 Introduction
Software testing is usually guided by the use of models that represent the system
under testing. These models provide information that supports such activities as test
case design, test verification and test coverage analysis. Depending on the nature of
the model used, software testing is called functional testing (also called specification-
based testing) or structural testing (also called implementation-based testing) [1].
Models used in functional testing are based on software functional requirements
while in structural testing are based on the software’s internal structure. Nowadays,
the most part of software testing is done using functional testing techniques, since
they are less costly. However, as research works have been focused on structural
testing, software industries do not have many formal ways to perform and assess the
test of their products and end up using just heuristics to carry out their software
testing [2].
The main goal of software testing is to detect defects as soon as possible during the
software development, looking for the improvement of the software quality. As
exhausting testing is impracticable, the application of testing criteria is desirable to
aid the selection of good test case sets, i.e., those that have a high chance to detect the
existent defects, but whose application is feasible. Such criteria can also be used in
analysis of the model coverage obtained by the application of test cases. In the former
case, the testing criteria are called selection criteria, and in the latter one, they are
called adequacy criteria [3], [4].
A functional testing technique that has been commonly used in the software
industry is Model Based Testing (MBT), which uses behavioral models of the
software, produced using its functional requirements, to carry out the software testing
[5], [6]. The modeling technique most used to create such models is the Finite State
Machines (FSMs). Besides software modeling, the FSMs are used as a basis to define
functional testing criteria to be applied in test case selection and adequacy analysis.
The FSMs are considered an excellent tool for software modeling, user/developer
communication and testing, since their application is simple and intuitive.
Although traditional FSMs applied in MBT have played an important role in
software testing improvement, they do not provide mechanisms to model important
behavioral aspects of the software such as its data flow. Due to this limitation, testing
techniques based just on traditional FSMs can not make use of further information.
For instance, existing functional testing criteria based only on the analysis of FSMs
control flow, demand just that elements such as states, arcs and loops are exercised.
As a result, the testing effort can achieve poor coverage of software functionality.
On the other hand, there are structural testing criteria based on both control flow
[1], [7] and data flow [3], [8] of the software’s internal structure, which is represented
by program graphs. Therefore, these testing criteria can provide several levels of the
internal structure coverage of the software being tested. Such criteria are widely used
in the code coverage analysis, during the execution of a test case set. Such analysis
counts on some support tools, such as POKE-TOOL, a software testing tool
developed at the State University of Campinas – UNICAMP [8], [9], [10].
In this paper, we propose a new set of functional testing criteria to be used in MBT
that considers data flow information of the software behavior. The proposed criteria
are based on an extension to traditional FSMs, which provides data flow modeling
mechanisms, and were defined extending know structural testing criteria, since the
extended FSMs (EFSMs) were defined structurally similar to program graphs.
Moreover, POKE-TOOL could also be used to support the application of the proposed
criteria. Despite the structurally similarity between the functional testing criteria and
the structural testing criteria, they are based on information of different nature and,
hence, their application can, for instance, lead to the detection of different kinds of
defects.
The application of the proposed criteria aims at providing greater quality in the test
of interactive systems, which answer to event occurrences, with data input – usually
provided by user’s actions, and are dominated by controls. Typical examples are
systems that interact with the user and whose execution flow depends on data
supplied by this interaction. The presented aproach mainly explore the user’s actions
during the system operation through data inputs and outputs. Other peculiarities of
interactive systems such as its concurrent nature or the importance of user tasks are
not take into account in that approach, being left to future work.
This paper is organized as follows. In Section 2, an overview is presented of MBT,
including related works. Section 3 contains the EFSMs definition. In Section 4, the
proposed functional testing criteria based on EFSMs are presented. Section 5 contains
the results of an experiment involving the application of the proposed functional
testing criteria and the structural testing criteria using POKE-TOOL. Finally, in
Section 5, the conclusions and future work are presented.
The following items are examples of simple functional criteria based on FSMs,
used to select test cases in the software industry:
• all-states: it requires that all states inside the FSM must be exercised by at least
one test case. Similarly to the structural testing criteria all-nodes, its satisfaction
offers a very poor coverage of the FSM;
• all-transitions: it requires that all transitions inside the FSM must be exercised
by at least one test case. Similarly to the structural testing criteria all-branches,
its satisfaction offers a poor coverage of the FSM;
• all-scenarios: it requires that all possible use scenarios must be exercised by at
least one test case. Similarly to the structural testing criteria all-paths, depending
on the FSM size, it can be considered impracticable.
As FSMs can be considered essentially graphs, many algorithms of graph theory
can be used in the implementation of these functional testing criteria, changing the
graph path concept by the FSM use scenario one. Examples of these algorithms are:
the Random Selection, the Chinese Postman and the Shortest Path First [2], [20], [21].
In addition to the criteria presented above, other criteria had been defined as results
of studies in the software testing field. The most known criteria are: Transition tour
[22]; W Method [23]; Wp Method [24]; Distinguishing Sequence Method DS [25],
[26], [27]; and Unique-Input-Output method [28]. Although the use of these criteria
offers an increased software quality and reliability assurance, they are more complex
and have a higher application cost. Thus, they are not usually used in the software
industry.
In this section, the extension to traditional FSMs, which provides data flow modeling
mechanisms, is presented. In addition to the elements of FSMs – such as states, inputs
and outputs, the proposed EFSMs contain other elements used for data flow
representation – such as variables, predicates and instructions. The EFSMs we
propose is intended to facilitate the definition of the functional testing criteria through
an extension of known structural testing criteria [1], [3], [8]. Moreover, the EFSMs
make possible the use of POKE-TOOL [9], [10] – a structural testing tool – to support
the functional testing criteria based on EFSMs.
An EFSM M is defined as an 11-tuple M = < S, s0, I, O, V, P, N, ρ, D, δ, λ, >,
where:
• S is a finite and non empty set of states;
• s0 ∈ S is the initial state;
• I is a finite set of inputs, including the null input;
• O is a finite set of outputs, including the null output;
• V is a finite set of variables;
• P is a finite set of predicates involving variables of V;
• N is a finite set of statements involving variables of V;
• ρ: SxVxN is the variable occurrence relation;
• D ⊆ SxIxP is the specification domain;
• δ: D → S is the state transition function;
• λ: D → O is the output function.
Each predicate p ∈ P can be formed by variables of V, constants and operators.
Possible operators are the logic operators AND, OR, and NOT, and the arithmetical
operators +, -, *, /, and =.
Each statement n ∈ N can be formed by variables of V, constants, operators,
commands and function calls. Possible operators are the arithmetical operators +, -, *
e / and the assignment operator ←. Possible commands are the data input command
READ and the data output command WRITE. Function calls are represented by fi(v1,
v2, ..., vn), where i, n ∈ (1, 2, 3, …) and v ∈ V.
There are three types of statements:
• data input statement – represented by READ(v), where v ∈ V;
• data output statement – represented by WRITE(v), where v ∈ V;
• data assignment statement – represented by v ← exp, where v ∈ V and exp is an
expression formed by variables of V, constants, operators and function calls.
A variable occurrence is represented by ρ(s, v, n). It means that there is an
occurrence of the variable v ∈ V, in the state s ∈ S, in the statement n ∈ N. There can
be, in a state s, more than one occurrence of the variable v, in different statements n.
A state transition is represented by δ(si, i, p) = sj and the output produced by the
execution of this transition is represented by λ(si, i, p) = o. It means that, when the
EFSM M is in a current state si ∈ S and receives an input i ∈ I, if the predicate p ∈ P
is satisfied, then it moves to a next state sj ∈ S, producing an output o ∈ O.
Variables and predicates are needed in EFSMs to represent the data flow associated
to the behavior of the modeled software. Probably, they are not the variables and the
predicates used internally in a program that implements the EFSM. Actually, the
EFSM variables and predicates are at a higher abstraction level than that of software
implementations. Thus, although possible, it is not necessary to keep a 1-to-1
mapping between the EFSM variables and predicates and that ones used in a program
that implements the EFSM.
An EFSM M can be deterministic or non deterministic. If |δ(s, i, p)| = 1 ∀s ∈ S, ∀i
∈ I and ∀p ∈ P, then the machine is deterministic, i.e., from a state s, on an input i
and if the predicate p is true, the machine follows a unique transition to a next state.
Otherwise, the machine is non deterministic, i.e., it is not possible to define
beforehand which transition must be followed and, hence, different outputs can be
produced.
Although the proposed criteria – EFSM-based criteria – are functional testing criteria,
they are derived from known data flow criteria of the structural testing [1], [3], [8].
4.1 Definitions
Definitions similar to the ones presented in [3], [8] are needed to explain the EFSM-
based criteria.
Variable occurrences in EFSMs can be of three types:
• Variable definition (referred as def): when the variable is part of a data input
statement – as READ(v), or is part of the left side of an assignment statement;
• Variable computational use (referred as c-use): when the variable is part of a
data output statement – as READ(v), or is part of the right side of an assignment
statement;
• Variable predicative use (referred as c-use): when the variable is part of a
predicate.
In a general way, a def variable occurs when a variable receives a value and a c-use
or p-use variable occurs when the variable value is being used. The c-use type affects
directly the computation being done or allows that the result obtained from a previous
variable definition can be observed. The p-use type affects directly the program
control flow.
According to the variable occurrence types, the following sets of variables are
defined:
• def(s): is the set of variables for which the state s contains a variable definition;
• c-use(s): is the set of variables for which the state s contains a variable c-use;
• p-use(s): is the set of variables for which the transition (si, sj) contains a variable
p-use.
Two types of scenarios used in the definitions of the EFSM-based criteria are:
• scenario: finite sequence of states (s1, ..., sk), k ≥ 2, such that there is a transition
from si to si + 1 for i = 1, 2, ..., k – 1;
1
• definition-free scenario: a definition-free scenario w.r.t. x from state i to state j
or to transition (sm, j) is a scenario (i, s1, ..., sm, j), including no def of x in states
s1, ..., sm.
Finally, the following sets are also needed. Let s be any state and x any variable
such that x ∈ def(s):
1
w.r.t. – with respect to
• dcu(x, s): is the set of all states si such that x ∈ c-use(si) and for which there is a
definition-free scenario w.r.t. x from s to si;
• dpu(x, s): is the set of all transitions (si, sj) such that x ∈ p-use(si, sj) and for
which there is a definition-free scenario w.r.t. x from s to si.
Although the application of the EFSM-based criteria aims at providing greater quality
in the test of interactive systems, the EFSMs can also be used to model the behavior
of non reactive systems – as traditional FSMs. Therefore, the use of the EFSM-based
criteria can be extended to test other types of system.
Compared to the already existent criteria based on traditional FSMs, the proposed
EFSM-based criteria here are more demanding. Thus, their satisfaction and the non
detection of defects, offers a greater quality and reliability indication for the tested
software.
The EFSM-based criteria are complementary to the structural testing criteria from
which they were derived. Therefore, their application can detect the presence of
different kinds of defects. For instance, the application of the EFSM-based criteria
can detect the absence of a non implemented functionality.
However, there is a cost related to the behavioral modeling using data flow
information through EFSMs. Considering the whole EFSM construction, the effort
can be too large; however, considering the effort already incurred into traditional
FSMs construction, the additional effort is affordable.
An experiment was carried out to experiment and validate the application of the
EFSM-based criteria. This experiment aimed at comparing the results of the
application of the proposed EFSM-based criteria and the application of the respective
structural testing criteria. In this experiment, the criteria are applied as adequacy
criteria, i.e., they are used to analyze the coverage obtained in the model with the
criteria application. No analysis of the type and amount of detected defects was done.
The EFSM-based criteria were implemented using TEST-TOOL, developed at the
author’s university [8], [9], [10].
POKE-TOOL is used to analyze the model coverage through the execution of a
program, i.e., to verify how much a set of test cases satisfies a certain testing criteria.
This tool was originally developed to support the application of structural testing
criteria, analyzing the coverage of models that represents the software internal
structure. Despite this, it was extended to support the application of the criteria
defined here, analyzing the coverage of EFSM behavior models.
The experiment was carried out in the following steps:
1. Elaborate the EFSM that represents the behavior of an interactive system whose
implementation is to be tested;
2. Implement a system that simulates the execution of the system modeled by the
elaborated EFSM;
3. Design and execute test cases until 100% coverage is reached for every EFSM-
based functional testing criterion used in this experiment;
4. Compare the amount of test cases needed to reach 100% coverage for each
EFSM-based criterion;
5. Execute the test cases designed in step 2 (used to reach 100% coverage for
EFSM-based criteria) considering the respective structural testing criteria;
6. Compare the reached coverage between the EFSM-based criteria and the
structural testing criteria upon the same test case set.
The modeled interactive system used in this experiment is a telecom application to
handle calls from a telephone-booth controlled by computer system. The user utilizes
the system to choose the payment type, inform the telephone number to be called,
make the call and, after the calling ends, obtain data concerning the call. Figure 2
presents a part of the modeled EFSM.
The EFSM-based criteria used were: two criteria for traditional FSMs – all-states
and all-transitions; and three of the criteria for EFSMs – all-p-uses, all-uses and all-
potential-uses.
The results obtained until step 4 of the experiment are presented in Table 1. These
results show the different exigency levels of the EFSM-based criteria for the
application modeled, implemented and tested. While just three test cases were enough
to exercise the 14 states in the EFSM, 23 test cases were needed to exercise all the
185 variable potential uses in the EFSM.
Table 1. Results from the application of EFSM-based criteria. Amount of required elements per
criterion and amount of test cases needed to reach 100% coverage of each criterion.
Criterion Required Elements Test Cases
all-states 14 3
all-transitions 17 6
all-p-uses 28 8
all-uses 48 8
all-potential-uses 185 23
These results do not mean that the most demanding criteria have a greater chance
of detecting defects. However, the most demanding criteria need a greater amount of
test cases to be completely satisfied, exercising more the software under test.
Therefore, they can be used as a parameter to determine how much the program was
tested.
To make clear, the EFSM scenarios exercised by the three test cases used to satisfy
completely the all-states criteria are presented below, using the number that represent
them (see Figure 2). The test cases used are not presented since some of them use
many input data and/or have many expected results.
• <1>, <2>, <3>, <5>, <6>, <8>, <9>, <10>, <12>, <14>;
• <1>, <2>, <5>, <7>, <12>, <13>, <2>, <14>;
• <1>, <2>, <4>, <5>, <6>, <8>, <9>, <10>, <11>, <12>, <14>.
Analyzing these three scenarios, we can see that they do not cover several elements
required by the other EFSM-based criteria, which are more demanding. For instance,
the transitions <7>, <6> and <13>, <5> are not exercised; the variable definition-use
associations from state <2> to transition <9>, <12> w.r.t. f_p and from state <3> to
transition <3>, <3> w.r.t. res are not exercised as well.
The results obtained from step 5 until step 6 of the experiment are presented in
Table 2. These results shows that, for the tested program, the EFSM-based criteria are
less demanding than the respective structural testing criteria, being, therefore, easier
to be satisfied. On an average, the test cases, whose execution reached 100% coverage
of EFSM-based criteria, reached 78% coverage of the structural testing criteria. Such
result is explained by the fact that a system implementation often needs to consider
more details than its own specification.
Table 2. Results from the structural testing criteria application. Amount of required elements
per criterion, amount of test cases executed (the same used in steps 3 and 4), and the reached
coverage for each criterion.
Criterion Required Elements Test Cases Reached Coverage
all-nodes 14 3 78%
all-branches 17 6 74%
all-p-uses 28 8 80%
all-uses 48 8 69%
all-potential-uses 185 23 81%
3.
Cancel Get Telephonic
Card Data res = T 5.
READ(n_c) Do
Present READ(pw) Phone Call
Wellcome 2. res ← f2(n_c, res = F
Message Determine f_p = calling_card READ(n_t)
pw)
Receive Payment tariff ←
Phone Call 4. f3(n_t)
Type res = F Cancel res = T
Get Credit
READ(f_p) Card Data Answer
Answer
1. READ(n_c)
f_p = cancel f_p = credit_card
Session res ← f1(n_c) f_p = collect_call
Begin op3 = T Present
!(f_p = collect_call)
f_p = collect_call Collect Call
Message
13. 6.
Change op3 = F Conversation
7.
Present Payment Receive o_r = T ti ← f4 ( )
Thanks Type? o_r = F
Message
Receptor
READ(op3) Confirmation Ring off
READ(o_r)
op2 = T
14.
Session 8.
12. Finish the
End op2 = F 9.
Do Call
f_p = collect_call
Store Data
Present Another 11. in Database tf ← f( )
Thanks Call? Show
Message DB ← f_p t = tf – ti
READ(op2) Call Data 10. DB ← n_c v = t * tariff
SHOW(n_t) Inform DB ← n_t
op1 = T
SHOW(t) Data about !(f_p = collect_call)
DB ← ti
SHOW(v) Done Call?
DB ← t
op1 = F READ(op1) DB ← v
Fig. 2. Part of the EFSM that represents the behavior of the telecom interactive application used in the experiment
6 Conclusion
References