FSM Testing Techniques

Introduction
Test from FSM

Complete Tests
Examples
Conclusions
Teste Baseado em Maquinas

de Estados Finitos
- SSC/ICMC/USP
Adenilso Simao
adenilso@icmc.usp.br
Departamento de Sistemas de Computacao
Instituto de Ciencias
Matematicas
e de Computacao
Paulo
Universidade de Sao
Arequipa - Peru - CLEI 2015 - 21-10-2015
- SSC/ICMC/USP adenilso@icmc.usp.br
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Software Testing
Software testing consists of the dynamic verification of the

behavior of a program on a finite set of test cases, suitably
selected from the usually infinite execution domain, against the
expected behavior.
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Software Testing (II)
Test Generation is Always Model-Based

I
Implicit models
Explicit models
I
I
System Understanding
Diagrams
State Machines
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
State Machines
Simplest explicit models

I
I
I
Vanilla models
Understandable for non-experts
Semantic is the model itself
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
State Machines (II)
Many flavors
I
State machine
I
LTS
IOTS
I
I
State and transitions between states

State machine, transitions labelled with actions
LTS, actions partitioned between inputs and outputs
FSM
I
LTS, transitions labelled with pair input/output
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Finite State Machine
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Test from State Machines
Given a specification machine
Given an implementation
Behaviour
Mainly input/output relation

I
Oracle problem
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Test from FSM
Long tradition
I
I
I
Moore, 19581
Hennie, 19652
Gonenc, 19703
1
Edward F. Moore. Gedanken-Experiments on Sequential Machines. In: J.
Symbolic Logic 23.1 (1958).
2
F. C. Hennie. Fault-detecting experiments for sequential circuits. In:
Proceedings of Fifth Annual Symposium on Circuit Theory and Logical Design. 1965,
pp. 95110.
3
G. Gonenc. A method for the design of fault detection experiments. In: IEEE
Transactions on Computers 19 (1970), pp. 551558.
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Test from FSM (II)

I
Active Area
I
Many recent papers

I
I
I
I
Hierons, 20134
Kapus-Kolar, 20125
El-Fakih, 20126
Simao, 20127
Robert M. Hierons. Verifying and Comparing Finite State Machines for Systems
that Have Distributed Interfaces. In: IEEE Trans. Computers 62.8 (2013),
pp. 16731683.
5
Monika Kapus-Kolar. On Exploring alternatives for transition verification. In:
Journal of Systems and Software 85.8 (2012), pp. 17441748.
6
Khaled El-Fakih et al. FSM-based testing from user defined faults adapted to
incremental and mutation testing. In: Programming and Computer Software 38.4
(2012), pp. 201209.
7
A. Petrenko, and N. Yevtushenko. On reducing test length for FSMs
A. Simao,
with extra states. In: Softw. Test., Verif. Reliab. 22.6 (2012), pp. 435454.
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Test from FSM (III)

I
Three best papers in 2012

I
I
I
ICST 20128
A-MOST 20129
SBQS 201210
8
and N. Yevtushenko. Generating Checking Sequences for
A. Petrenko, A. Simao,
Nondeterministic Finite State Machines. In: ICST. ed. by Giuliano Antoniol,
Antonia Bertolino, and Yvan Labiche. IEEE, 2012, pp. 310319. ISBN:
978-1-4577-1906-6.
9
Experimental Comparison of Test Case Generation
A. Endo and A. Simao.
Methods for Finite State Machines. In: ICST. ed. by Giuliano Antoniol,
Antonia Bertolino, and Yvan Labiche. IEEE, 2012, pp. 549558. ISBN:
978-1-4577-1906-6.
10
M. L. Capellari et al. Towards Incremental FSM-based Testing of Software
Product Lines. In: Simposio

Brasileiro de Qualidade de Software. Fortaleza, Ceara,
2012, pp. 923.
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Test from FSM (IV)
Coverage based
I
Cover models elements

I
I
States
Transitions
Fault based
I
I
Mutants
Fault domains
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Fault domains
Set of faulty FSM

I
Each representing a kind of fault

I
I
I
I
Output faults
Transfer faults
Extra states
Missing states
(It can be thought of as) a huge set of mutants

I O ((n k )(nl ) ), where n is the number of states, k is the number
of inputs, and l is the number of outputs
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Fault domains (II)
Given an FSM M with n states

I (M ) is the set of all FSM with at most m states
m
It is assumed that the implementation can be modeled by some

N m (M ), for some known m
I
Testing Hypothesis
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Mutant Killing
Transition Tour
I
x1 x0 x1 y1 x1 y0 y0 y1 y0
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Mutant Killing (II)
Transition Tour
I
I
x1 x0 x1 y1 x1 y0 y0 y1 y0
to kill it: y0 y1
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Mutant Killing (III)
Transition Tour
I
I
I
x1 x0 x1 y1 x1 y0 y0 y1 y0
y0 y1
to kill it: x1 y0 y0 y1 y0 x1 x0 x1
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Mutant Killing (IV)
Transition Tour
I
I
I
I
x1 x0 x1 y1 x1 y0 y0 y1 y0
y0 y1
x1 y0 y0 y1 y0 x1 x0 x1
to kill it: y0 x1 x1
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Mutant Killing (V)
Transition Tour
I
I
I
I
I
x1 x0 x1 y1 x1 y0 y0 y1 y0
y0 y1
x1 y0 y0 y1 y0 x1 x0 x1
y0 x1 x1
to kill it: y0 y1 x0
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Mutant Killing (VI)
There are (n k )(nl ) = (4 2)(42) = 16.777.216 mutants
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Fault Domain Based
Distinguishing Sequence (DS): yyy

I
I
I
I
for s1, y0 y1 y0
for s2, y1 y0 y0
for s3, y0 y0 y1
for s4, y0 y0 y0
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Fault Domain Based
Verify each transition

I
I
Transfer to the starting state, Apply the DS

Transfer to the starting state, Apply the input, Apply the DS
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Fault Domain Based
Consider (s4, x)
I

I
y0 y1 y0 y0 y0

I
y0 y1 x1 y0 y0 y1
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Fault Domain Based
Consider (s2, x)
I

I
y0 y1 y0 y0

I
y0 x1 y1 y0 y0
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Fault Domain Based
Complete test suite (W Method):

I
I
I
I
I
x1 y0 y0 y1 y0
x1 x0 y1 y0 y0
y0 x1 y1 y0 y0
y0 y1 x1 y1 y0 y0
y0 y1 y0 y0 y0 y0
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Fault Domain Based
Complete test suite (Checking Sequence):

I
y0 y1 y0 y0 y0 y0 x1 y0 y0 y1 x1 x0 y1 x1 x0 x1 y1 x1 y0 x1 y0 y0
y1
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Fault domains
Fault Domain Based
Complete test suite (Perebor / Brute Force):

I
y0 x1 y1 y0 y0 y0 x1 y0 x1 y0 y0 y1 x1 x0 y1
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Generating Complete Tests
Complete Tests
Given a specification M and a fault domain m (M )

I A test suite T is complete (w.r.t. M and m (M )), if
I
Each N m (M ) which is not equivalent to M is killed by some test

t T
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Complete Tests (II)
Suppose that we have a complete test suite T and that the

implementation can be modeled by some N m (M )
I
I
Either a fault is revealed by T ;

Or N is equivalent to M
I
The implementation (which is modeled by N) is correct
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Several Cases
M is deterministic or not?
M is minimal or not?
M is complete or not?
N m (M ) is deterministic or not?
m < n, m = n or m > n?
The implementation has a reliable reset or not?
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Required properties
M should be initially connected
N m (M ) should accept at least the inputs of M
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Many methods
I
Depending on the case: deterministic, minimal, complete, etc.
Basic idea
I
I
Recognize the states of the implementation

Verify the transitions of the implementation
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Reliable Operations
I
Magic operations
I
Set
I
State Recognition
I
I
Reset
Status
Set operation
Status operation
Transition Verification
I
I
I
Set operation
Execute the transition
Status operation
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Without Reliable Operations
Instead of Set
I
Transfer sequences
Instead of Status
I
Separating sequences
I
Many possibilities
Instead of Reset
I
Not many options
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
W Method (1978)
Separating Sequence
I
Given a pair of states, a Separating Sequence for those states is

an input sequence for each both states produce different outputs
Characterization Sets W
I
I
Contains separating sequences for each pair of states

Can replace Status Operations
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
HSI Method (1997)
Harmonized State Identifiers

I
A set Hs for each state s
For each pair of states s and t, there is an input sequence in

Pref (Hs ) Pref (Ht ) which separates s and t
I Pref (A) is the of all prefixes of sequences in A
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
H Method (2005)
Separating sequences are generated on-the-fly

I
I
More expensive
Shorter tests
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
SPY Method (2011)
Avoiding branches
I
I
Consider the test aaaaa is already in the test suite

It is possible to choose between two tests for a given purpose
I
I
aab
aaaaab
Which is better?
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Checking Sequence
When resetting is not an option

I
I
Locating Sequence
I
Impossible
Too expensive
Exponentially long
Distinguishing Sequence
I
A singleton characterization set
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions
Conclusions
Long-standing problem
I
Several results
Open problems
I
I
I
I
Non-determinism
Decoupling input and outputs
Necessary conditions
More efficient methods
Adenilso Simao

de Estados Finitos
Introduction
Test from FSM
Complete Tests
Examples
Conclusions

de Estados Finitos
- SSC/ICMC/USP
Adenilso Simao
adenilso@icmc.usp.br
Departamento de Sistemas de Computacao
Instituto de Ciencias
Matematicas
e de Computacao
Paulo
Universidade de Sao
Arequipa - Peru - CLEI 2015 - 21-10-2015
Adenilso Simao

de Estados Finitos

FSM Testing Techniques

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

FSM Testing Techniques

Uploaded by

Copyright:

Available Formats

Introduction

Test from FSM

Teste Baseado em Maquinas

Departamento de Sistemas de Computacao

Arequipa - Peru - CLEI 2015 - 21-10-2015

Teste Baseado em Maquinas

Software testing consists of the dynamic verification of the

Teste Baseado em Maquinas

Software Testing (II)

Test Generation is Always Model-Based

Teste Baseado em Maquinas

Simplest explicit models

Teste Baseado em Maquinas

State Machines (II)

State and transitions between states

LTS, transitions labelled with pair input/output

Teste Baseado em Maquinas

Finite State Machine

Teste Baseado em Maquinas

Test from State Machines

Given a specification machine

Mainly input/output relation

Teste Baseado em Maquinas

Test from FSM

Teste Baseado em Maquinas

Test from FSM (II)

Many recent papers

Teste Baseado em Maquinas

Test from FSM (III)

Three best papers in 2012

Product Lines. In: Simposio

Teste Baseado em Maquinas

Test from FSM (IV)

Cover models elements

Teste Baseado em Maquinas

Set of faulty FSM

Each representing a kind of fault

(It can be thought of as) a huge set of mutants

Teste Baseado em Maquinas

Fault domains (II)

Given an FSM M with n states

It is assumed that the implementation can be modeled by some

Teste Baseado em Maquinas

Teste Baseado em Maquinas

Mutant Killing (II)

Teste Baseado em Maquinas

Mutant Killing (III)

Teste Baseado em Maquinas

Mutant Killing (IV)

Teste Baseado em Maquinas

Mutant Killing (V)

Teste Baseado em Maquinas

Mutant Killing (VI)

There are (n k )(nl ) = (4 2)(42) = 16.777.216 mutants

Teste Baseado em Maquinas

Fault Domain Based

Distinguishing Sequence (DS): yyy

Teste Baseado em Maquinas

Fault Domain Based

Verify each transition

Transfer to the starting state, Apply the DS

Teste Baseado em Maquinas