BITS Pilani

presentation
BITS Pilani Vivek Kulkarni
Guest Faculty
Pilani Campus
 BITS Pilani
Pilani Campus

<SEZG552/SSZG552, STM>
Lecture No.1
Introduction

Testing has two fundamental purposes:

1. To evaluate quality or acceptability of that which is

being tested.

2. To discover problems or errors.

BITS Pilani, Pilani Campus

Further expanded into 3 groups of objectives

Demonstration Detection Prevention

Show that the system can be Discover defects, errors, and Provide information to
used with acceptable risk. deficiencies Prevent or reduce the
Number of errors

Show that products are Determine system capabilities Reduce the number of early
ready and limitations errors propagated through
for integration or use. to later phases

Demonstrate functions under Determine quality of Clarify system

special conditions components, work products, specifications
and the system and performance
Identify ways to avoid risks
And problems in the future.
Basic definitions

Error (mistakes) - we call them BUGS

Fault ( defect) – result of an error
Failure – occurs when faults execute
Incident – symptom of failure
Test- is obviously concerned with all the above and
is the art of exercising software with test cases
Test case- has an identity and is associated with
program behavior

BITS Pilani, Pilani Campus

Taxonomy -- Boris Bezier

Mild: e.g., a misspelled output.

Moderate: e.g., bug impacts performance.
Annoying: e.g., bills for $0.00 are sent.
Disturbing: e.g., ATM won’t give you
money because of bug.
Serious: e.g., Your paycheck transaction is
not recorded.
Very Serious: e.g., Your paycheck is
deposited to the wrong account.
Catastrophic: e.g., Bug starts a war.

BITS Pilani, Pilani Campus

BITS Pilani, Pilani Campus
 testing
normal robust

weak strong weak strong

 Equivalence Class Testing

• Use the mathematical concept of

partitioning into equivalence classes
to generate test cases for Functional
(Black-box) testing

• The key goals for equivalence class

testing are similar to partitioning:
1. completeness of test coverage
2. lessen duplication of test coverage
 Weak Normal Equivalence testing

1. Assumes the ‘single fault’ or “independence

of input variables.”
– e.g. If there are 2 input variables, these input
variables are independent of each other.

2. Partition the test cases of each input

variable separately into different equivalent
classes.

3. Choose the test case from each of the

equivalence classes for each input variable
independently of the other input variable
 Example of : Weak Normal Equivalence testing
Assume the equivalence partitioning of input X is: 1 to 10; 11 to 20, 21 to 30
and the equivalence partitioning of input Y is: 1 to 5; 6 to 10; 11;15; and 16 to 20

X We have covered everyone

of the 3 equivalence classes
30 for input X.

20

For ( x, y )
10 we have:
(24, 2)
(15, 8 )
1 ( 4, 13)
Y (23, 17)
1 5 10 15 20 General rule for # of test cases?
What do you think?
We have covered each of the 4
# of partitions of the largest set?
equivalence classes for input Y.
 Strong Normal Equivalence testing

• This is the same as the weak normal

equivalence testing except for
“multiple fault assumption”
or
“dependence among the inputs”

• All the combinations of equivalence classes

of the variables must be included.
 Example of : Strong Normal Equivalence
testing
Assume the equivalence partitioning of input X is: 1 to 10; 11 to 20, 21 to 30
and the equivalence partitioning of input Y is: 1 to 5; 6 to 10; 11;15; and 16 to 20

30

We have covered everyone

of the 3 x 4 Cartesian
20 product of equivalence
classes

10

1
Y
1 5 10 15 20
General rule for # of test cases?
What do you think?
 Weak Robust Equivalence testing
• Up to now we have only considered
partitioning the valid input space.

• “Weak robust” is similar to “weak

normal” equivalence test except that
the invalid input variables are now
considered.

A note about considering invalid input is that there may not

be any definition “specified” for the various, different invalid
inputs - - - making definition of the output for these invalid
inputs a bit difficult at times. (but fertile ground for testing)
 Example of : Weak Robust Equivalence testing
Assume the equivalence partitioning of input X is 1 to 10; 11 to 20, 21 to 30
and the equivalence partitioning of input Y is 1 to 5; 6 to 10; 11;15; and 16 to 20

30

We have covered
20 everyone of the 5
equivalence classes
for input X.
10

1
Y
1 5 10 15 20

We have covered each of the 6

equivalence classes for input Y.
 Strong Robust Equivalence testing

• Does not assume “single fault” - - - assumes

dependency of input variables

• “Strong robust” is similar to “strong normal”

equivalence test except that the invalid input
variables are now considered.
 Example of : Strong Robust Equivalence
testing
Assume the equivalence partitioning of input X is: 1 to 10; 11 to 20, 21 to 30
and the equivalence partitioning of input Y is: 1 to 5; 6 to 10; 11;15; and 16 to 20

30
We have covered everyone
of the 5 x 6 Cartesian
product of equivalence
20 classes (including invalid
inputs)

10

1
Y
1 5 10 15 20
Boundary Value Analysis
• F(x1, x2), a ≤ x1 ≤ b, c ≤ x2 ≤ d
• Boundary value analysis focuses on the boundary
of the input space to identify test cases
• The rationale behind boundary value testing is that
errors tend to occur near the extreme values of an
input variable
• e.g. loop conditions (< instead of ≤), counters

1
Boundary Value Analysis
• Basic idea: use input variable values at their
minimum (min), just above the minimum (min+), a
nominal value (nom), just below their maximum
(max-), and at their maximum (max)
• “Single fault” assumption in reliability theory:
failures are only rarely the result of the
simultaneous occurrence of two (or more) faults.
• The boundary value analysis test cases are obtained
by holding the values of all but one variable at their
nominal values, and letting that variable assume its
extreme values

2
 Boundary Value Analysis
F(x1, x2)
x2 Input Space (domain) x2 Test Cases (function
of two variables)
<x1nom, x2min>
<x1nom, x2min+>
d d <x1nom, x2nom>
<x1nom, x2max->
<x1nom, x2max>
<x1min, x2nom>
<x1min+, x2nom>
<x1nom, x2nom>
<x1max-, x2nom>
c c <x1max, x2nom>

x1 x1
a b a b

3
Generalizing Boundary Value Analysis

• By the number of variables

• For n variables → 4n + 1 test cases
• By the kinds of ranges, depends on the type (nature) of the
variables
• Variables have discrete, bounded values
• e.g. NextDate function, commission problem
• Variables have no explicit bounds
• Create “artificial” bounds
• e.g. triangle problem
• Boolean variables
• Decision table-based testing
• Logical variables (bound to a value or another logic variable)
• e.g. PIN and transaction type in SATM System

4
Limitations of Boundary Value Analysis

• Boundary value analysis works well when the

program to be tested is a function of several
independent variables that represent bounded
physical quantities
• e.g. NextDate test cases are inadequate (little stress on
February, dependencies among month, day, and year)
• e.g. variables refer to physical quantities, such as
temperature, air speed, load etc.

5
Robustness Testing
• Simple extension of boundary value analysis
• In addition to the five boundary value analysis
values of a variable, see what happens when the
extrema are exceeded with a value slightly greater
than the maximum (max+) and a value slightly less
than the minimum (min-)
• Focuses on the expected outputs
• e.g. exceeding load capacity of a public elevator
• Forces attention on exception handling

6
Robustness Testing
x2 Robustness Test Cases
<x1nom, x2min>
(function of two variables) <x1nom, x2min+>
<x1nom, x2nom>
<x1nom, x2max->
d <x1nom, x2max>
<x1min, x2nom>
<x1min+, x2nom>
<x1nom, x2nom>
<x1max-, x2nom>
<x1max, x2nom>
<x1min-, x2nom>
<x1nom, x2max+>
c <x1max+, x2nom>
<x1nom, x2min->

x1
a b

7
Worst Case Testing
• Worst case analysis: more than one variable has an extreme value
• Procedure:
• For each variable create the set <min, min+, nom, max-, max>
• Take the Cartesian product of these sets to generate test cases
• More thorough than boundary value analysis
• Represents more effort
• For n variables → 5n test cases (as opposed to 4n+1 test cases for
boundary value analysis)
• Follows the generalization pattern
• Same limitations
• Robust worst case testing can be applied

8
Worst Case Testing
x2 Worst Case Test <x1min, x2min> <x1max-, x2min>
Cases (function of two <x1min, x2min+> <x1max-, x2min+>
variables) <x1min, x2nom> <x1max-, x2nom>
<x1min, x2max-> <x1max-, x2max->
d <x1min, x2max> <x1max-, x2max>
<x1min+, x2min> <x1max, x2min>
<x1min+, x2min+> <x1max, x2min+>
<x1min+, x2nom> <x1max, x2nom>
<x1min+, x2max-> <x1max, x2max->
<x1min+, x2max> <x1max, x2max>
<x1nom, x2min>
c <x1nom, x2min+>
<x1nom, x2nom>
x1 <x1nom, x2max->
a b
<x1nom, x2max>

9
 Worst Case Testing
x2 Robust Worst Case Test
<x1min-, x2min-> <x1min+, x2min-> <x1max-, x2min-> <x1max+, x2min->
Cases (function of two
<x1min-, x2min> <x1min+, x2min> <x1max-, x2min> <x1max+, x2min>
variables)
<x1min-, x2min+> <x1min+, x2min+> <x1max-, x2min+> <x1max+, x2min+>
<x1min-, x2nom> <x1min+, x2nom> <x1max-, x2nom> <x1max+, x2nom>
d
<x1min-, x2max-> <x1min+, x2max-> <x1max-, x2max-> <x1max+, x2max->
<x1min-, x2max> <x1min+, x2max> <x1max-, x2max> <x1max+, x2max>
<x1min-, x2max+> <x1min+, x2max+> <x1max-, x2max+> <x1max+, x2max+>
<x1min, x2min-> <x1nom, x2min-> <x1max, x2min->
<x1min, x2min> <x1nom, x2min> <x1max, x2min>
<x1min, x2min+> <x1nom, x2min+> <x1max, x2min+>
<x1min, x2nom> <x1nom, x2nom> <x1max, x2nom>
c
<x1min, x2max-> <x1nom, x2max-> <x1max, x2max->
<x1min, x2max> <x1nom, x2max> <x1max, x2max>
x1
a b <x1min, x2max+> <x1nom, x2max+> <x1max, x2max+>

10
Decision Table Based Testing

Neelam Gupta

The University of Arizona

Tucson, Arizona, USA
 Decision Tables
Stub Entry
C1 T F

Condition C2 T F T F

C3 T F - T F -
a1 x x x
a2 x x
Action a3 x xx
a4 x x

Rule
 Test Case Generation
• Conditions: inputs
• Actions: outputs
• Rules: test cases
C1: a,b,c form a N Y
triangle
C2: a=b? - Y N

C3: a=c? - Y N Y N

C4: b=c? - Y N Y N Y N Y N

A1: not a triangle x

A2: Scalene x
A3: Isosceles x x x
A4: Equilateral x
A5: Impossible x x x
Triangle Problem: accept 3 integers as input
Output: Equilateral, Isosceles, Scalene,
NotATriangle.
 Expand c1: a,b,c are a triangle to
a<b+c? b<a+c? c<a+b?

C1: a<b+c? N - - Y Y Y Y Y Y Y Y
C2: b<a+c? N - Y Y Y Y Y Y Y Y
C3: c<a+b? N Y Y Y Y Y Y Y Y
C4: a=b? Y Y Y Y N N N N
C5: a=c? Y Y N N Y Y N N
C6: b=c? Y N Y N Y N Y N

A1: not a triangle x x x

A2: Scalene x
A3: Isosceles x x x
A4: Equilateral x
A5: Impossible x x x
When conditions refer to equivalence classes,
decision tables have characteristic
appearance.
R1 R2 R3
C1: month in M1 T - -
C2: month in M2 - T -
C3: month in M3 - - T
a1
a2
a3

Rule count 2 for each don’t care

4+4+4=12
Since we should have only 8 rules, where is the
problem?
Condition 1.1 1.2 1.3 1.4 2.1 2.2 2.3 2.4 3.1 3.2 3.3 3.4

C1: M1 T T T T T T F F T T F F
C2: M2 T T F F T T T T T F T F
C3: M3 T F T F T F T F T T T T
Rule count 1 1 1 1 1 1 1 1 1 1 1 1

Delete repetitions, missing rule is in which

all conditions are false.
Condition 1.1 1.2 1.3 1.4 2.3 2.4 3.4

C1: M1 T T T T F F F F
C2: M2 T T F F T T F F
C3: M3 T F T F T F T F
a1: invalid input x x x x x
a2:
a3:
 The ability to recognize and develop complete
decision tables makes them strong with respect
to redundancy and inconsistency.
Condition 1-4 5 6 7 8 9

C1 T F F F F T
C2 - T T F F F
C3 - T F T F F
a1 x x x - - -
a2 - x x x - x
a3 x - x x x -

Rules 4 and 9 are inconsistent because action

Sets are different. The whole table is non-deterministic.
Conclusion: Be careful with don’t care entries used in a
decision table.
Let us consider the decision table for next date
problem with the following equivalence classes.
M1 = {month: month has 30 days}
M2 = {month: month has 31 days}
M3 = {month: month is February}
D1 = {day: 1<=day<=28}
D2 = {day: day=29}
D3 = {day: day=30}
D4 = {day: day=31}
Y1 = {year: year = 1900}
Y2 = {year: 1812<=year<=2012 AND (year!=1900)
AND(year=0 mod 4)}
Y3 = {year: (1812<=year<=2012 AND year!=0 mod 4}
Conditions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

C1: month in M1 M1 M1 M1 M2 M2 M2 M2 M3 M3 M3 M3 M3 M3 M3 M3

C2: day in D1 D2 D3 D4 D1 D2 D3 D4 D1 D1 D1 D2 D2 D2 D3 D3

C3: year in - - - - - - - - Y1 Y2 Y3 Y1 Y2 Y3 - -

Rule Count 3 3 3 3 3 3 3 3 1 1 1 1 1 1 3 3
Actions

a1: impossible x x x x x
a2: increment day x x x x x ? x ?
a3: reset day x x ? ? x
a4: increment month x ? ? ? x
a5: reset month ?
a6: increment year ?
To resolve the ? entries, we need to refine the
equivalence classes. Revised equivalence classes for
the Next Date function are:
M1 = {month: month has 30 days}
M2 = {month: month has 31 days except December}
M3 = {month: month is December}
M4 = {month: month is February}
D1 = {day: 1<=day<=27}
D2 = {day: day=28}
D3 = {day: day=29}
D4 = {day: day=30}
D5 = {day: day=31}
Y1 = {year: year = 1900}
Y2 = {year: 1812<=year<=2012 AND (year!=1900)
AND(year=0 mod 4)}
Y3 = {year: (1812<=year<=2012 AND year!=0 mod 4}
Cause-Effect Graphing
 Cause-Effect Graphing is a test case design technique that is
performed once requirements have been reviewed for
ambiguity, followed by a review for content. Requirements
are reviewed for content to insure that they are correct and
complete.

 The Cause-Effect Graphing technique derives the minimum

number of test cases to cover 100% of the functional
requirements to improve the quality of test coverage.
Cause-Effect graph
 Captures the relationships between specific
combinations of inputs (causes) and outputs
(effects)
 Deals with specific cases,
 Avoids combinatorial explosion
 Explore combinations of possible inputs
Test Case Review
Once the test cases have been derived by the test case designer, the RBT process
includes the review of these test cases by the following stakeholders:

• The author of the requirements verifies that he/she agrees with the test case
designer’s
translation of requirements to test cases.
• The domain experts review the test cases in order to determine the answer to the
following question: “Are we building the right system”?
• The developers review the test cases to clarify their understanding of the
requirements. The developers learn what they will be tested on, and can therefore
develop the software to succeed.
By performing these reviews, everyone on the project team can obtain the same
understanding of what will be built.
Instead of the test case designer trying to manually determine
the right set of test cases, he/she models the problem using a
cause-effect graph, and the software that supports the
technique, BenderRBT, calculates the right set of test cases to
cover 100% of the functionality.

The cause-effect graphing technique uses the same algorithms

that are used in hardware logic circuit testing. Test case design
in hardware insures virtually
defect free hardware.
The Cause-Effect Graphing technique begins with the set of
requirements, and determines the minimum number of test
cases to completely cover the requirements.
Any type of logic can be modeled using a Cause-Effect
diagram. Each cause (or input) in the requirements is
expressed in the cause-effect graph as a condition, which is
either true or false.
.
Each effect (or output) is expressed as a condition, which is
either true or false
 I have a requirement that says: “If A OR B, then C.”

The following rules hold for this requirement:

• If A is true and B is true, then C is true.
• If A is true and B is false, then C is true.
• If A is false and B is true, then C is true.
• If A is false and B is false, then C is false.
All requirements are translated into nodes and relationships on the cause-effect
graph. There are only four possible relationships among nodes, and they are
indicated by the following symbols:

Where A always leads to C, a straight line ------------.

Where A or B lead to C, a V at the intersection means “or”.
Where A and B lead to C, an inverted V at the intersection means “and”.
A tilde ~ means “not” as in “If not A, then C”.
Drawing Cause-Effect Graphs

A B
If A then B
(identity)
A
C
B
If (A and B)then C
Drawing Cause-Effect Graphs

A
C
B
If (A or B) then C
A
 C
B
If (not(A and B)) then C
Drawing Cause-Effect Graphs

A
 C
B
If (not (A or B))then C

A  B
If (not A) then B
 Constraint Symbols

E: at most, one of a
and b can be 1
a

b
a
O
O: Exactly one of A and B
can be invoked
b
a
I: at least one of a or b
must be 1
b
Example: ATM
 For a simple ATM banking transaction system
 Causes (inputs)
 C1: Command is credit
 C2: command is debit
 C3: account number is valid
 C4: transaction amount is valid
 Effects
 E1: Print “invalid command”
 E2: Print “ invalid account number”
 E3: Print “debit amount not valid”
 E4: debit account
 E5: Credit account
 
or E1

C1
and
 E2

C2
and E3

C3
and E4

C4
and E5
Description of processing nodes
used in a Cause-effect graph
Type of processing description
node
AND Effect occurs if all input
are true (1)
OR Effect occurs if both or
one input is true
XOR Effect occurs if one of
input is true
Negation () Effect occurs if input
are false (0)
ATM Cause-effect decision table
Don’t Care condition
Cause\effect R1 R2 R3 R4 R5
C1 0 1 x x 1
C2 0 x 1 1 x
C3 x 0 1 0 1
C4 x x 0 1 1
E1 1 0 0 0 0
E2 0 1 0 0 0
E3 0 0 1 0 0
E4 0 0 0 1 0
E5 0 0 0 0 1
More Example
 Consider the following NL specifications:
 The boiler should be shut down if any of the following
conditions occurs:
 If the water level in a boiler is below the 20000lb mark
 The water level is above the 120000 lb, or
 The boiler is operating in degraded mode and the steam meter fails
 Being in degraded mode means :If either water pump or a pump monitors fails
Example 3: Translate NL into workable pieces
(atomic specifications)
 Atomic sentences are
 C1. the water level in boiler is below the 20000 lb (F/T)
 C2. the water level is above the 120000 lb mark (F/T)
 C3. A water pump has failed (F/T)
 C4. A pump monitor has failed (F/T)
 C5. Steam meter has failed/T
 E. Shut the boiler down
Steps to create cause-effect graph
1. Study the functional requirements and divide the requirements into workable pieces
1. E.g. of workable piece, in eCom, can be verifying a single item placed in shopping cart
2. Identify causes and effects in the specifications
1. Causes: distinct input conditions
2. Effects: an output condition or a system transformations.
3. Assign unique number to each cause and effect
4. Use the semantic content of the spec and transform it into a Boolean graph
5. Annotate the graph with constrains describing combinations of causes and/or effects
6. Convert the graph into a limited-entry decision table
7. Use each column as a test case
Possible research topics based on CEG
 Comparison of CEG ,FSM-based test sets, and randomly
generated test cases (functional)
 For effectiveness in terms of cost and fault detection capabilities
 For fault detection capabilities
 For number of test cases generated (cost)
 For automatic generation of actual test cases
Code Base Testing
(white box)
 What is White-box Testing?
• Looking at the internal structure of a program
and deriving test cases based on the logic or
control flow.
• Test cases can be designed to reach every
branch in the code and to exercise each
condition
• Typically done during unit testing

• Also known as:

– Structural Testing
– Glass-Box Testing
 What is Black-box Testing?
• Looking at the program from an external
point of view and deriving test cases
based on the specification.
• The only criteria upon which the program
is judged is if it produces the correct
output for a given input.
 Why Do Both?
Black-box White-box
• Impossible to write a test • Does not address the
case for every possible question of whether or
set of inputs and outputs not the program matches
• Some of the code may the specification
not be reachable without • Does not tell you if all of
extraordinary measures the functionality has been
• Specifications are not implemented
always complete • Does not discover
missing program logic
 Unit test tools
• AdaTEST • HtmlUnit
• AQtest • HttpUnit
• ObjcUnit • SUnit
• Aunit • JavaScript
• OCUnit • TagUnit
• C++Test • JsUnit
• OTF - An • TBGEN
• Cantata • JsUnit
• PalmUnit • TBrun
• Check • JTestCase •
• PBUnit Test Mentor
• COMPUTE • JUnit - Java
• PerlUnit
• CppUnit • JUnitEE Edition
• phpAsserU•
• csUnit • JUnitX nit unit++
• CTA++ • LingoUnit • PhpUnit • vbUnit3
Basic
• CTB • MinUnit • PyUnit
• cUnit • Mock Creator• • VectorCAS
QtUnit T
• CUT • Mock Objects• Ruby/Mock• XMLUnit
• dotunit • MockMaker
• XSLTunit
• EasyMock • Mockry
• •
GrandTestAuto NUnit
• HarnessIt
 Basic Program Flow Controls
• IF
• IF-Then-Else
• FOR
• While
• Do-While
• Case
IF Diagram

If
IF-THEN-ELSE Diagram

If-then-else
FOR or WHILE Diagram

For OR While
DO-WHILE Diagram

Do-while
CASE Diagram

Case
Example Code Fragment
Do
{
if (A) then {...};
else {
if (B) then {
if (C) then {...};
else {…}
}
else if (D) then {...};
else {...};
}
}
While (E);
Example Control Flow Graph

True

True
B

C D
True True

E
True
 White-box Test Methods
• Statement Coverage
• Decision/Branch Coverage
• Condition Coverage
• Decision/Condition Coverage
• Path Coverage
 Type of coverage
• Statement coverage
• Basic block coverage
• Decision coverage
• Condition coverage
• Branch coverage
• Loop coverage
 Test/Code coverage –
Statement
examples
Decision Path coverage

? ? ?

1 2 4 test cases
 Example Code Fragment
• If ((A>1) & (B=0))
then Do;
• X=X/A;
• END;
• If ((A==2) | (X>1))
then Do;
• X=X+1;
• END;
• END;
 Statement Coverage
• Exercise all
statements at least
once
• How many test
cases?
 A=2 and B=0 (ace)
 Decision/Branch Coverage
• Each decision has a
true and a false
outcome at least once
• How many test
cases?
 A=2 and B=0 (ace)
 A=1 and X=1 (abd)
 Condition Coverage
• Each condition in a
decision takes on all
possible outcomes at
least once
• Conditions: A>1, B=0,
A=2, X>1
• How many test cases?
 A=2, B=0, and X=4 (ace)
 A=1, B=1, and X=1 (abd)
 Decision/Condition Coverage
• Each condition in a decision
takes on all possible
outcomes at least once, and
each decision takes on all
possible outcomes at least
once
• How many test cases?
 A=2, B=0, and X=4 (ace)
 A=1, B=1, and X=1 (abd)
• What about these?
 A=1, B=0, and X=3 (abe)
 A=2, B=1, and X=1 (abe)
 Multiple Condition Coverage
• Exercise all possible
combinations of
condition outcomes in
each decision
• Conditions:
A>1, B=0 A=2, X>1
A>1, B<>0 A=2, X<=1
A<=1, B=0 A<>2, X>1
A<=1, B<>0 A<>2, X<=1
 Multiple Condition Coverage
• How many test
cases?
 A=2, B=0, X=4 (ace)
 A=2, B=1, X=1 (abe)
 A=1, B=0, X=2 (abe)
 A=1, B=1, X=1 (abd)
 Path Coverage
• Every unique path
through the program
is executed at least
once
• How many test
cases?
 A=2, B=0, X=4 (ace)
 A=2, B=1, X=1 (abe)
 A=3, B=0, X=1 (acd)
 A=1, B=1, X=1 (abd)
t f

f
f

f
Program
 Statement Coverage
• Reports on the percentage of executable
statements of the code exercised by a set
of test cases
• Cover all statements (cover all nodes)
• TC1: {a=8,b=7,c=6,max=1,count=1}
– P1:
(4,5,6,7,8,9,10,11,12,13,15,16,17,19,21,22,
23,25,26,7,27)
• TC2: {a=6,b=6,c=2,max=1,count=1}
– P2:
(4,5,6,7,8,9,10,11,12,13,15,16,17,18,19,21,
23,25,26,7,27)
• TC3: {a=3,b=3,c=3,max=1,count=1}
– P3:
(4,5,6,7,8,9,10,11,12,13,15,16,17,19,20,21,
23,25,26,7,27)
• TC4: {a=2,b=3,c=9,max=1,count=1}
– P4:
(4,5,6,7,8,9,10,11,12,14,15,24,25,26,7,27)
 Decision Coverage
• Reports whether boolean expressions in
control structures are evaluated to both true
and false values by the test cases
• TC1: {a=8,b=7,c=6,max=1,count=1}
– P1:
(4,5,6,7,8,9,10,11,12,13,15,16,17,19,21,22,
23,25,26,7,27)
• TC2: {a=6,b=6,c=2,max=1,count=1}
– P2:
(4,5,6,7,8,9,10,11,12,13,15,16,17,18,19,21,
23,25,26,7,27)
• TC3: {a=3,b=3,c=3,max=1,count=1}
– P3:
(4,5,6,7,8,9,10,11,12,13,15,16,17,19,20,21,
23,25,26,7,27)
• TC4: {a=2,b=3,c=9,max=1,count=1}
– P4:
(4,5,6,7,8,9,10,11,12,14,15,24,25,26,7,27)
• TC5: {a=1,b=1,c=1,max=10,count=100}
– P5: (4,5,6,7,27)
 Condition Coverage
• Reports the true or false
outcome of each
boolean sub-expression
• TC1:
{a=8,b=7,c=6,max=1,
count=1}
• TC2:
{a=3,b=3,c=3,max=1,
count=1}
• TC3: {a=2,b=3,c=9,
max=-2,count=-1}
• TC4: {a=9,b=3,c=2,
max=-2,count=-1}
• TC5: {a=3,b=9,c=2,
max=-2,count=-1}
 Condition/Decision Coverage
• The Union of Condition and Decision Coverage
• TC1: {a=8,b=7,c=6,max=1,count=1}
• TC2: {a=6,b=6,c=2,max=1,count=1}
• TC3: {a=3,b=3,c=3,max=1,count=1}
• TC4: {a=2,b=3,c=9,max=1,count=1}
• TC5: {a=1,b=1,c=1,max=10,count=100}
• TC6: {a=2,b=3,c=9, max=-2,count=-1}
• TC7: {a=9,b=3,c=2, max=-2,count=-1}
• TC8: {a=3,b=9,c=2, max=-2,count=-1}
 Modified Condition/Decision
Coverage
• Reports the true or false outcome of
each boolean sub-expression and
the true or false outcome of each
boolean expression containing them
• TC1: {a=8,b=7,c=6,max=1,count=1}
• TC2: {a=6,b=6,c=2,max=1,count=1}
• TC3: {a=3,b=3,c=3,max=1,count=1}
• TC4: {a=2,b=3,c=9,max=1,count=1}
• TC5:
{a=1,b=1,c=1,max=10,count=100}
• TC6: {a=2,b=3,c=9,max=-2,
count=-1}
• TC7: {a=9,b=3,c=2,max=-2,
count=-1}
• TC8: {a=3,b=9,c=2,max=-2,
count=-1}
 Path Coverage
• Reports whether each of the possible paths in
each function have been followed
• TC1: {a=8,b=7,c=6,max=1,count=1}
– P1: (4,5,6,7,8,9,10,11,12,13,15,16,17,19,21,22,
23,25,26,7,27)
• TC2: {a=6,b=6,c=2,max=1,count=1}
– P2: (4,5,6,7,8,9,10,11,12,13,15,16,17,18,19,21,
23,25,26,7,27)
• TC3: {a=3,b=3,c=3,max=1,count=1}
– P3: (4,5,6,7,8,9,10,11,12,13,15,16,17,19,20,21,
23,25,26,7,27)
• TC4: {a=2,b=3,c=9,max=1,count=1}
– P4: (4,5,6,7,8,9,10,11,12,14,15,24,25,26,7,27)
• TC5: {a=1,b=1,c=1,max=10,count=100}
– P5: (4,5,6,7,27)
• TC6: {a=2,b=3,c=9,max=2,count=1},
{a=2,b=3,c=9}
– P6:
(4,5,6,7,8,9,10,11,12,14,15,24,25,26,7,8,9,10,11,12,
14,15,24,25,26,7,27)
• The presence of the loop leads to an unlimited
number of paths, thus it is suggested we should
put an upper bound on the number of times the
loop is executed (e.g. max=10)
Example 2 (Decision Coverage)
• P1: first, A, B, D, F, G, last
• P2: first, A, B, C, D, F, G,
last
• P3: first, A, B, C, F, G, last
• P4: first, A, B, E, D, F, G,
last
• P5: first, A, B, E, F, G, last
• P6: first, A, B, D, F, G, A,
B, D, F, G, last
 Example 2
• Suppose we take the graph as the
DD-Path graph of some program.
Develop sets of paths (which would
be test cases) for the Statement
Coverage, Decision Coverage, and
Loop Coverage metrics
• Loop coverage reports
whether each loop body in the
program has been executed
zero times, exactly once, and
more than once
(consecutively)
 Example 2
• Statement Coverage
– P1: first, A, B, C, D, F, G, A, B, E, F, G, last
• Decision Coverage
– P1: first, A, B, D, F, G, last
– P2: first, A, B, C, D, F, G, last
– P3: first, A, B, C, F, G, last
– P4: first, A, B, E, D, F, G, last
– P5: first, A, B, E, F, G, last
– P6: first, A, B, D, F, G, A, B, D, F, G, last
• Loop coverage
– P1: first, A, B, D, F, G, last (loop not entered)
– P2: first, A, B, D, F, G, A, B, D, F, G, last (loop
entered once)
– P3: repeat P2 for n times (loop executed n
times)
– P4: repeat P2 for max times (loop executed
maximum times)
 McCabe’s Cyclomatic Complexity
• Software metric
• Developed by Tom McCabe (circa 1976)
• Directly measures the number of linearly
independent paths through a program’s
source code, taking into account the
various decision points
• Independent of implementation language
 Calculating Complexity

(a) # regions + 1 A

(b) # decisions + 1 True

(c) L – N + 2P True
B

C D
True True

E
True
 Calculating Complexity

(a) # regions + 1 = 5+1 = 6

A
(b) # decisions + 1 = 5+1 = 6
(c) L – N + 2P = 17-13+2 = 6 True

True
B

C D
True True

E
True
How Complex Should Code Be?
• <10: Simple module, not much risk
• 10-20: More Complex; moderate risk
• 20-50: Complex; high risk
• >50: Untestable; extremely high risk
 Complexity Caveats
• As code is broken into smaller modules to
decrease cyclomatic complexity, structural
complexity increases
• Some modules may have high complexity
but are very easy to comprehend and easy
to test
• High complexity numbers are only an
indicator of something to investigate
 Agenda
 Define/Use Testing Definitions
 Define/Use Testing Example
 Commission Problem
 DU-Path Test Coverage Metrics

1
 Agenda
 Define/Use Testing Definitions
 Define/Use Testing Example
 Commission Problem
 DU-Path Test Coverage Metrics

2
 Define/Use Testing Definitions
 Node n  G ( p ) is a defining node of the variable v  V,
written as DEF(v,n), iff the value of the variable v is
defined at the statement fragment corresponding to node
n
 Node n  G ( p ) is a usage node of the variable v  V,
written as USE(v,n), iff the value of the variable v is used
at the statement fragment corresponding to node n
 A usage node USE(v,n) is a predicate use (denoted as
P-use, outdegree ≥ 2) iff the statement n is a predicate
statement; otherwise USE(v,n) is a computation use
(denoted C-use, outdegree ≤ 1)

3
 Define/Use Testing Definitions
 A definition-use (sub)path with respect to a variable v
(denoted du-path) is a (sub)path in PATHS(P) such that,
for some v  V, there are define and use nodes
DEF(v,m) and USE(v,n) such that m and n are the initial
and final nodes of the (sub)path
 A definition-clear (sub)path with respect to a variable v
(denoted dc-path) is a definition-use (sub)path in
PATHS(P) with initial and final nodes DEF(v,m) and
USE(v,n) such that no other node in the (sub)path is a
defining node of v
 DU-paths that are not definition-clear are potential
trouble paths

4
 Agenda
 Define/Use Testing Definitions
 Define/Use Testing Example
 Commission Problem
 DU-Path Test Coverage Metrics

5
 Define/Use Testing Example

 Commission Problem
 Computes the commission on the sales of
four salespersons based on the total number
of locks, stocks and barrels sold by each one
of them

6
Define/Use Testing Example

7
Define/Use Testing Example

DD-PATH Nodes
1 14
2 15-22
3 23
4 24-30
5 31-39
6 40-44
7 45
8 46-49
9 50
10 51,52
11 53

DD-Paths

Program Graph DD-Path Graph

8
Define/Use Testing Example
Variable Defined at Used at Comment
locks 9 (to compiler)
locks 22 READ
locks 23 predicate use
locks 26 computation use
locks 29 READ
stocks 9 (to compiler)
stocks 25 READ
stocks 27 computation use
num_locks 9 (to compiler)
num_locks 19 assignment
num_locks 26 assignment
num_locks 26 computation use
num_locks 33 WRITE
num_locks 36 computation use
sales 11 (to compiler)
sales 36 assignment
sales 37 WRITE
sales 39 predicate use
sales 43 computation use
sales 45 predicate use
sales 48 computation use
sales 50 computation use
commission 11 (to compiler)
commission 41 assignment
commission 42 assignment
commission 42 computation use
commission 43 assignment
commission 43 computation use
commission 47 assignment
commission 48 assignment
commission 48 computation use
commission 50 assignment
commission 51 WRITE
Define/Usage Nodes for locks, stocks, num_locks,
sales and commission variables
9
 Define/Use Testing Example

 We use the defining and usage nodes in

conjunction with the program graph to
identify various definition-use and
definition-clear paths
 It’s a judgment call whether or not non-
executable statements such as constant
(CONST) and variable (VAR) declaration
statements should be considered as
defining nodes

10
Define/Use Testing Example
Variable Defined at Used at Comment
locks 9 (to compiler)
locks 22 READ
locks 23 predicate use
locks 26 computation use
locks 29 READ
stocks 9 (to compiler)
stocks 25 READ
stocks 27 computation use
num_locks 9 (to compiler)
num_locks 19 assignment
num_locks 26 assignment
num_locks 26 computation use
num_locks 33 WRITE
num_locks 36 computation use
sales 11 (to compiler)
sales 36 assignment
sales 37 WRITE
sales 39 predicate use
sales 43 computation use
sales 45 predicate use
sales 48 computation use
sales 50 computation use
commission 11 (to compiler)
commission 41 assignment
commission 42 assignment
commission 42 computation use
commission 43 assignment
commission 43 computation use
commission 47 assignment
commission 48 assignment
commission 48 computation use
commission 50 assignment
commission 51 WRITE
Define/Usage Nodes for locks, stocks, num_locks,
sales and commission variables
11
 Variable
Define/Use Testing Example
Defined at Used at Comment Du-Path Variable Def Node Use Node
locks 9 (to compiler) 1 locks 22 23
locks 22 READ 2 locks 22 26
locks 23 predicate use 3 locks 29 23
locks 26 computation use 4 locks 29 26 Du-Path Variable Def Node Use Node
locks 29 READ 5 stocks 25 27 27 barrels 25 28
stocks 9 (to compiler) 6 num_locks 19 26
stocks 25 READ
28 num_stocks 20 27
7 num_locks 19 33 29 num_stocks 20 34
stocks 27 computation use
8 num_locks 19 36 30 num_stocks 20 36
num_locks 9 (to compiler)
9 num_locks 26 33 31 num_stocks 27 34
num_locks 19 assignment
num_locks 26 assignment
10 num_locks 26 36 32 num_stocks 27 36
num_locks 26 computation use 11 sales 36 37 33 num_barrels 21 28
num_locks 33 WRITE 12 sales 36 39 34 num_stocks 21 35
num_locks 36 computation use 13 sales 36 43 35 num_stocks 21 36
sales 11 (to compiler) 14 sales 36 45 36 num_stocks 28 35
sales 36 assignment 15 sales 36 48 37 num_stocks 28 36
sales 37 WRITE 16 sales 36 50
sales 39 predicate use 17 commission 41 42 Du-Paths B’
sales 43 computation use 18 commission 41 43
sales 45 predicate use 19 commission 41 51
sales 48 computation use
sales 50 computation use
20 commission 42 43 DU-paths such as <26,26>
21 commission 42 51
commission 11 (to compiler) 22 commission 43 51 are disallowed
commission 41 assignment 23 commission 47 48
commission 42 assignment 24 commission 47 51
commission 42 computation use
25 commission 48 51
commission 43 assignment
26 commission 50 51
commission 43 computation use
commission 47 assignment Du-Paths A’
commission 48 assignment
commission 48 computation use
commission 50 assignment
commission 51 WRITE
Define/Usage Nodes for locks, stocks, num_locks,
sales and commission variables
12
 Agenda
 Define/Use Testing Definitions
 Define/Use Testing Example
 Commission Problem
 DU-Path Test Coverage Metrics

13
 DU-Path Test Coverage Metrics
 The set T satisfies the All-Defs criterion for the program
P iff for every variable v  V, T contains definition clear
(sub)paths from every defining node of v to a use of v
 The set T satisfies the All-Uses criterion for the program
P iff for every variable v  V, T contains definition-clear
(sub)paths from every defining node of v to every use of
v, and to the successor node of each USE(v,n)
 The set T satisfies the All-P-Uses/Some C-Uses criterion
for the program P iff for every variable v  V , T contains
definition-clear (sub)paths from every defining node of v
to every predicate use of v, and if a definition of v has no
P-uses, there is a definition-clear path to at least one
computation use

14
 DU-Path Test Coverage Metrics
 The set T satisfies the All-C-Uses/Some P-Uses criterion
for the program P iff for every variable v  V, T contains
definition-clear (sub)paths from every defining node of v
to every computation use of v, and if a definition of v has
no C-uses, there is a definition-clear path to at least one
predicate use
 The set T satisfies the All-DU-paths criterion for the
program P iff for every variable v  V, T contains
definition-clear (sub)paths from every defining node of v
to every use of v, and to the successor node of each
USE(v,n), and that these paths are either single loop
traversals, or they are cycle free

15
 DU-Path Test Coverage Metrics

 Example: All-Uses criterion wrt locks

Du-Path Variable Def Node Use Node Use Node Type
1 locks 22 23 predicate use
2 locks 22 26 computation use
3 locks 29 23 predicate use
4 locks 29 26 computation use

 All-C-Uses
 <22,26>, <29,26>
 All-P-Uses
 <22,23>, <29,23>
 All-Uses = All-C-Uses Union All-P-Uses
 <22,26>, <29,26>, <22,23>, <29,23>

16
 DU-Path Test Coverage Metrics
 When one test coverage metric
subsumes another, a set of test cases
that attains coverage in terms of the
first metric necessarily attains coverage
with respect to the subsumed metric
 Since several du-paths are present in a
full program execution path (traversed
by a test case), the higher forms of
coverage metrics don’t always imply
significantly higher numbers of test
cases
Case locks stocks barrels sales commission DD-Paths
1 5 5 5 500 50 1-5, 7, 9, 10, 11
2 15 15 15 1500 175 1-5, 7, 8, 10, 11
3 25 25 25 2500 360 1-5, 6, 10, 11

 The missing du-paths (8, 9, 14, 18, 19)

are all traversed by a test case in which
nothing is sold (i.e., the first value of
locks is -1)
Hierarchy of Data Flow
Coverage Metrics

17
McCabe basis path

18
19
20
t f

f
f

21
Basis Path Testing

22
23
ex1= p2+p3-p1
Ex1= (1,0,1,2,0,0,0,0,1,0) +(1,0,0,0,1,0,0,1,0,1)-(1,0,0,1,0,0,0,0,1,0)
Ex1 = (1,0,1,1,1,0,0,1,0,1)=A,B,C,B,E,F,G
ex2=2p2-p1
Ex2=(2,0,2,4,0,0,0,0,2,0) – (1,0,0,1,0,0,0,0,1,0)= (1,0,2,3,0,0,0,0,1,0)=A,B,C,B,C,B,C,G

24
 McCabe’s Algorithmic
Procedure
 McCable’s Algorithmic Procedure can be
used to identify a set of basis paths
 Select a path with highest number of decision
nodes
 Retrace each decision in baseline path
 Flip each decision to create a new path
 E.g.
 P1: A,B,C,B,E,F,G
 A,D,E,F,G (flip p1 at A)
 A,,B,E,F,G (flip p1 at B)
 A,B,C,G (flip p1 at C)

25
Observations on McCabe’s Basis
Path
Is testing Basis
path enough? Infeasible paths

26
 Essential Complexity
V(G)=1wellStr(P)
V(G) >=10 requires
substantial testing
Use the characteristics
of code to select
appropriate criteria

27
28