Quality Assurance and Software Testing

Overview

1 White box testing

Control flow testing
Data flow testing
Fault-based testing
Quality Assurance and Software Testing
White box testing

White box testing

Takes into account the internal mechanism of a system or

component [IEEE 1990]
Also called structural testing, clear box or glass box testing
White box can be used in 3 types of testing: unit testing,
integration testing and regression testing
Quality Assurance and Software Testing
White box testing

White box testing of code

The software under test (SUT)

implementation is analysed
Paths through SUT are identified
Inputs are chosen to cause the SUT to
execute selected paths=path
sensitization
Expected outputs are determined
Actual outputs are compared with the
expected outputs
A conclusion is made regarding the
proper functioning of the SUT
Quality Assurance and Software Testing
White box testing

White box testing

Each test case’s inputs activate a certain path in the SUT

Quality Assurance and Software Testing
White box testing

How to design test cases?

It depends on metrics
Control flow testing
Coverage testing-uses coverage metrics
Basis path testing-uses complexity metrics
Boundary values and equivalence classes
Data flow testing
Quality Assurance and Software Testing
White box testing

Control flow testing

Identifies execution paths through a module

Generates test cases to force the program to follow these paths
Based on the assumption that more bugs will be detected if
more paths will be executed.
Ideal testing is 100% path testing
Quality Assurance and Software Testing
White box testing

Disadvantages

Cannot test paths that were not implemented

Ideal testing is 100% path testing
A module can execute correctly for 99% of cases and fail for a
few

double r a t i o ( i n t a , i n t b) {
r e t u r n ( a /b ) ;
}

Control flow may be correct but processing may be incorrect

a=a+1 // a c t u a l b u t i n c o r r e c t c o d e
a=a−1 ; // c o r r e c t c o d e
Quality Assurance and Software Testing
White box testing

Control flow graph

Quality Assurance and Software Testing
White box testing

Control flow graph

Nodes= statements/fragments
Edges show the flow of control
Decision points
Junction points
Only one entry at the beginning
of graph
Only one exit at the end
Quality Assurance and Software Testing
White box testing

Example

i f ( a>0)
x=x +1;
i f ( b==3)
y =0;
Quality Assurance and Software Testing
White box testing

Testing metrics

Testing process management needs a measure of the extend

of testing – testing metrics
White box testing uses different metrics for test evaluation.
One of them is the test coverage metrics.
Quality Assurance and Software Testing
White box testing

Coverage

Coverage is what people invented instead of exhaustive testing

Coverage is a measure of completeness of the set of test cases
and shows how much code has been exercised during testing.
There are hundreds types of testing coverages (loops,
exceptions, lines of code, branches, etc) [Kaner]
Coverage analysis can identify code that has not been
executed.
Quality Assurance and Software Testing
White box testing

Levels of coverage

Level 0. Test whatever you test. Let the users test the rest:
irresponsible
Level 1. 100% statement coverage = during testing the flow
of control reached each executable line of code at least one.
Quality Assurance and Software Testing
White box testing

Level 1

Level 1: 100% Statement coverage

Quality Assurance and Software Testing
White box testing

Statement coverage evaluation

Weak, because bugs are sensitive to branches and often

boundary conditions like (x¡3) are wrongly implemented
The logic can be wrong
Does not test a simple if without else
But even this coverage is in practice difficult to achieve so it is
better than nothing
Quality Assurance and Software Testing
White box testing

Level 2: 100% Decision coverage

Also called branch

coverage. For
each decision
point both
branches True
and False are
exercised.
Guarantees
statement
coverage
Quality Assurance and Software Testing
White box testing

Level 3: Condition coverage

Each condition in each decision is evaluated at least once.

i f ( a>0 && c==1){

x=x +1;
}

Test cases for 100% condition coverage

a=3; c=1;
a=0; c=2;
Quality Assurance and Software Testing
White box testing

Level 4: Decision/condition coverage

Test cases are created for each condition and each decision

i f ( ( x>1) && ( y >10)){

a =2;
}

Test cases
x=2 y=11
x=2 y=9
x=0 y=11
x=0 y=9
Quality Assurance and Software Testing
White box testing

Level 5:Loop coverage

First test case: execute a loop zero times

Second test case: execute a loop 1 time
Third test case: execute a loop n times
Fourth test case execute the loop the max. nr of times, m.
In addition m-1, m+1 times
Quality Assurance and Software Testing
White box testing

Level 6: 100% path coverage

Equal to exhaustive white box testing, usually impossible

Quality Assurance and Software Testing
White box testing

How to design test cases?

Analyzes the topology of control flow graph to determine the

test cases
Ensures that all linearly independent paths have been tested
An independent path transverses at least one edge that no
other path does.
Guarantees both statement and decision coverage.
Quality Assurance and Software Testing
White box testing

McCabe complexity

The McCabe cyclomatic complexity (C) measures the number

of linearly independent paths through a program module

C = e-n+2

e = nr. of edges
n=number of nodes
Quality Assurance and Software Testing
White box testing

McCabe complexity metrics

Provides a stronger measure of a module’s structural

complexity than by counting lines of code
The number of tests required for a software module is equal
to its cyclomatic complexity [from NIST Structured testing]
1-10: simple program, 21-50: high risk program, > 50
untestable program
Quality Assurance and Software Testing
White box testing

Test cases design

Derive the control flow graph for the tested module

Compute the cyclomatic complexity C
Select a set of C basis paths
Create a test case for each basis path
Quality Assurance and Software Testing
White box testing

Algorithm for basis path testing

Pick a baseline path ( a typical

path of execution)
Change the outcome of the first
decision while keeping the
maximum of the rest unchanged
Begin again from the baseline
but change the outcome of the
second decision
etc.
Quality Assurance and Software Testing
White box testing

Example
Quality Assurance and Software Testing
White box testing

Basis path testing-evaluation

Basis path testing –cyclomatic complexity is the lower

boundary of how much testing is necessary
Path testing provides a metrics that act as a cross checks on
functional testing. Can signal redundancies in black box test
cases. If they exercise the same path, the redundancy does
not reveal new faults.
Other techniques have to be used in addition
Quality Assurance and Software Testing
White box testing

Unit testing

Unit test – a piece of code T that exercises a unit U and

determines whether U is behaving correctly (how the
programmer expected).
During executing T you actually run the code U under
carefully controlled conditions
Mock objects- if the method under test needs some external
data sources, like a database, they are simulated using mock
objects
Quality Assurance and Software Testing
White box testing

Unit test tools

We will practice with JUnit – a very popular open source

framework for Java unit testing.
The unit tests are written in Java
http://www.junit.org
Can run independently or as a plug-in for Eclipse IDE
Quality Assurance and Software Testing
White box testing

Assertions

How to check if code behaves as you expect?

Use assertions-help methods that verify that something is
true.
Assertions in JUnit: a line of code that describes what you
expect and let JUnit compare your expectation against the
actual result. Ex : assertEquals(expected, actual)
Quality Assurance and Software Testing
White box testing

The test class

A test class extends the TestCase base class

A test class contains test methods
Test methods don’t have parameters
Each test method contains one or more assertions
Quality Assurance and Software Testing
White box testing

JUnit naming conventions

Conventions, and not JUnit requirements

The test class named after the class under test, ex:
AccountTest if the class under test is Account
Name of class is important – should be of the form
TestMyClass or MyClassTest
This naming convention lets TestRunner automatically find
your test classes
Quality Assurance and Software Testing
White box testing

JUnit Assertions

Assertions
assertEquals(expected, actual)
assertEquals(expected, actual, delta) for floats Ex:
assertEquals(3.33, 10.0/3.0, 0.01)
assertEquals(message,expected,actual)
assertTrue(condition)
Can be found in JUnit API
Quality Assurance and Software Testing
White box testing

A simple example

Assertions
public class Calculator {
p u b l i c s t a t i c i n t add ( i n t a , i n t b ){
r e t u r n ( a+b ) ;
}
}

Is tested by the JUnit test case CalculatorTest.java:

import j u n i t . framework . TestCase ;

p u b l i c c l a s s C a l c u l a t o r T e s t extends TestCase {
p u b l i c v o i d testAddTwoAndThree ( ) {
a s s e r t E q u a l s ( 5 , C a l c u l a t o r . add ( 2 , 3 ) ) ;
}
}

A test method passed if all its assertions are true.

Quality Assurance and Software Testing
White box testing

Coverage analysis

dJUnit is a coverage reporting tool

an Eclipse plugin
Generates a coverage report with statement and decision
coverage in %.
Quality Assurance and Software Testing
White box testing

Data flow testing

Is a powerful tool to detect this type of errors

#i n c l u d e <s t d i o . h>
main ( ) {
int x ;
i f ( x==42) { d o s t u f f ( ) }

Based on the observation that computing the wrong value

leads to a failure only when the value is subsequently used.
Focus is moved from control flow to data flow
Used in compilers design
In testing help select paths that have higher chance to
propagate the wrongly computed data to a point of
observable failure
Quality Assurance and Software Testing
White box testing

Variable’s life cycle

Variables have a life cycle: created, used, destroyed.

In some languages (Java, C) var need declaration before use,
in other languages (Perl, Javascript, Python) the variables are
created during the first assignment = dynamic typing.
Data flow testing looks at the life cycle of each variable in the
module and explores strange things that can happen to data
Quality Assurance and Software Testing
White box testing

Data flow graph

Is similar to a control flow graph, it shows the processing flow

in a module, but
In addition it contains data flow information (annotations)
about definition, use and destruction of each variable in the
module
Quality Assurance and Software Testing
White box testing

Annotations

d -definition – Ex: input statements, assignments,loop control

assignments, procedure calls (statements where the contents
of the memory location changes)
u -use (statements where memory location is not changed)
p – use: used in a predicate
c –use: used for computation
Quality Assurance and Software Testing
White box testing

Def-use (DU) pair

Is a pair of statements, one is a definition occurrence and the

other is a use occurrence of a variable.
Value of x at 6 could be
computed at 1 or at 4
Bad computation at 1 or
4 could be revealed only
if they are used at 6
(1, 6) and (4, 6) are
def-use(DU) pairs
defs at 1,4
use at 6
Quality Assurance and Software Testing
White box testing

Def-use (DU) path

Definition-clear path = a path without any other definition

occurrence of the same variable
Def-use (DU) path is a definition-clear path between a def
and a use statement, a channel for data flow.
Quality Assurance and Software Testing
White box testing

Example

For variable x :
def in 1 and 4
c-use in 4
p-use in 5
DU paths:
1 − 2 − 3 − 5 and
1−2−4
Quality Assurance and Software Testing
White box testing

Data flow based test coverage criteria

All-defs
All DU pairs (All c-uses All p-uses)
All DU paths
Quality Assurance and Software Testing
White box testing

All-defs coverage

at least one definition-clear path from every definition to some

corresponding use is executed
The all defs coverage of a test set T for a program P is :
Cdefs =number of covered definitions/number of definitions
Quality Assurance and Software Testing
White box testing

All p-uses, c uses

All p-uses
Ensures that if a variable is defined and later used within a
conditional expression, this def-use pair is executed at least once
causing the surrounding decision to evaluate true, and once
causing the decision to evaluate false

All c-uses
Ensures that if a variable is defined and used later, at least one
definition clear path between these 2 events will be executed.
Quality Assurance and Software Testing
White box testing

All DU pairs

Also called All-uses

Covers all p-uses and all c-uses
All DU coverage guarantees statement and branch coverage,
but has a finer grain coverage. Needs for example a while loop
to be executed twice,control flow testing only once.
Quality Assurance and Software Testing
White box testing

All DU-paths

Test cases must exercise all DU paths for each definition. If

there are multiple path from one def to a use, all paths must
be executed.
Usually a reasonable amount, but sometimes nr. of test cases
can grow exponentially.
Quality Assurance and Software Testing
White box testing

Test coverage criteria

Impose an obligation for test specification

Can be used to generate a test specification or to measure
how good a test set is.
Test coverage criterion A subsumes test coverage B iff for
every program P, every test satisfying A also satisfies B
Quality Assurance and Software Testing
White box testing

Test coverage

Test coverage criteria hierarchy (control flow and data flow based)
Quality Assurance and Software Testing
White box testing

How to design test cases

Draw first the control flow graph

Draw based on this control flow graph for each variable a
data flow graph.
Identify paths according to the chosen coverage criterion.
Design test cases to sensitize the selected paths
Quality Assurance and Software Testing
White box testing

Data flow testing evaluation

Data flow testing should be used after control flow testing

Data flow testing helps to reduce the nr. of test cases
generated by control flow testing
Leads to more efficient and targeted test suites
Quality Assurance and Software Testing
White box testing

Fault based testing

Can be used with two goals:

To measure test case’s adequacy. The goal is to answer the

question : how good is our test case set?
To generate test cases. The goal is to produce a test set that
differentiates the program from each of its alternates.
Quality Assurance and Software Testing
White box testing

Fault based testing

How good is a test set?

In fault-based testing, the software under test is modified and the
test set is run on this modified version in order to determine the
test set adequacy.
Quality Assurance and Software Testing
White box testing

Fault based testing

Fault based testing techniques

Mutation testing
Fault injection
Quality Assurance and Software Testing
White box testing

Fault based testing

Mutation testing assumptions

A program is well tested if a majority of simple faults are

detected and removed.
Competent programmer hypothesis, programs are near-perfect,
errors are small deviations from intended program
Coupling effect hypothesis, simple and complex errors are
coupled
Quality Assurance and Software Testing
White box testing

Fault based testing

The procedure
Simple faults are introduced into the SUT by creating a set of
faulty versions, named mutants. Each mutant has a small
change and is created by a mutation operator, described by
syntactic changes.
The test set is run on the mutants and the result is compared
with the original results
From this comparison the tester draws conclusions and
improves his testing- indirectly mutation testing is a method
to design test cases
Quality Assurance and Software Testing
White box testing

Fault based testing

Mutation testing
Select mutation operators
Generate mutants
Distinguish mutants
Quality Assurance and Software Testing
White box testing

Fault based testing

Mutation operators
Example
AOR- arithmetic operator replacement Applied to an assignment
statement result = a + b Yields 4 mutants:

result=ab
result =ab
result=ab
result=a%b
Quality Assurance and Software Testing
White box testing

Fault based testing

Equivalent mutant
Equivalent mutant differs syntactically, but semantically is
the same with the original program
A problem for mutation testing because and it can never be
killed
Open research question: detecting equivalent mutants
Quality Assurance and Software Testing
White box testing

Fault based testing

Original The equivalent mutant

y =2; y =2;
x=y ∗ 2 ; x=y +2;
i f ( x<z ) i f ( x<z )
x =1; x =1;
e l s e i f ( x==z ) e l s e i f ( x==z )
x =0; x =0;
e l s e x=x +1; e l s e x=x +1;
p r i n t f ( ”%d” , x ) ; p r i n t f ( ”%d” , x ) ;
Quality Assurance and Software Testing
White box testing

Procedure

Equivalent mutant
Test set run on the mutant gives another output, fault is
detected, the mutant is declared killed by the test.
The output is the same: it can be that the mutant is
equivalent. If it is not, then the mutant is declared alive. The
test case failed to detect the fault – it is ineffective and new
test cases have to be designed.
In this way, mutation testing helps tester to start a new
iteration and improve his testing.
Quality Assurance and Software Testing
White box testing

Fault based testing

Original A mutant
y =2; y =2;
x=y ∗ 2 ; x=y ∗ 4 ;
i f ( x<z ) i f ( x<z )
x =1; x =1;
e l s e i f ( x==z ) e l s e i f ( x==z )
x =0; x =0;
e l s e x=x +1; e l s e x=x +1;
p r i n t f ( ”%d” , x ) ; p r i n t f ( ”%d” , x ) ;

Test case z=5 kills the mutant

Test case z=10 fails to kill the mutant
Quality Assurance and Software Testing
White box testing

Mutation score

Equivalent mutant
Given a SUT and a test case set, the method returns a value,
called mutation score (MS)
MS=(#killed mutants/total-#equivalent mutants)x100%
Test is mutation adequate if the mutation score is 100%
Quality Assurance and Software Testing
White box testing

Practice

In practice it is difficult to determine which mutant is

equivalent (automatic tools exist)
It is acceptable to calculate MS as:
MS=(#killed mutants/totalx100%
The higher the score the more effective is the test case at
fault detection
Quality Assurance and Software Testing
White box testing

Mutation testing evaluation

Method computationally expensive involves many executions,

it costs a lot of time and money
Has been too slow for practical adoption
Mutation automatic tools are needed
Only simple faults can be introduced. Not based on field data.
Special problems are introduced by OOP
Quality Assurance and Software Testing
White box testing

Mutation tools

Performs mutations at bytecode level.

Has 3 major functions:
Generate mutants for traditional mutation operators and for
class-level mutation operator
Run test cases supplied by the tester
Evaluates the mutation coverage of the tests
Quality Assurance and Software Testing
White box testing

MuClipse

Muclipse is the reincarnation of MuJava mutation tool as an

Eclipse plugin
Found on sourceforge
2 runtime configuations
Generate mutants
Runs a given test case on the original code and on the mutants
and compares the results
Quality Assurance and Software Testing
White box testing

Fault injection

Fault are injected as close as possible to the fault that can

occur in the field and check how the system reacts.
Used to test safety critical systems (space shuttle, railway
control, nuclear power plants, fly-by-wire, medical life- keeping
devices)
Inject hardware faults -no disk, take out a chip, etc.
A recent approach – emulate faults by software
Quality Assurance and Software Testing
White box testing

Fault injection

Problems
The injected fault has to be representative of software faults
observed in the field
Characterization and classification of faults (defects) is an
essential step in fault injection
The time issue