CS3346

Software Testing and Maintenance

Combinatorial Testing

Combinatorial_s
 Contents
• Application examples
• The default value strategy
• Single vs multiple mode faults
• Pairwise coverage
• The combinatorial design approach
• The In-Parameter-Order (IPO) algorithm
• Beyond pairwise coverage: The IPOG algorithm
• Reading, Tool and Reference
Example: Component-Based Systems
Payment Gateway
Alipay, Amazon,
PayPal, …

Client Browser Web Server Database System

Chrome, Edge, Apache, Google GWS, Microsoft SQL, MySQL,
Firefox, ... Microsoft IIS, … Oracle, …

• A system can be built from components as above.

How do we ensure they interoperate properly?
 Example: Character Formatting
• A presentation software allows several choices of
parameters when formatting a character.
– Its typeface can be chosen from Arial, Comic,
Courier or Times (default).
– Its style can be regular (default), bold, italic or
boldItalic.
– It can be underlined or notUnderlined (default).
– Its font size can be 10, 11 (default) or 12.
• How to test that the characters will be formatted
in the proper typeface, style, underlining & size?
 Example: The Telephone Call Problem
Example: Consider the problem of testing a telephone
switch’s ability to place telephone calls.
The system has 4 parameters , each 3 values :
Call Type Billing Access Status
Local Caller Residential Success
Long Distance Collect Commercial Busy
International Free Dedicated Blocked

Q: How many tests have to be done to fully test the

various values of the parameters?
A: _______________________________________
 A Naïve but Systematic Strategy
Very often, each parameter has a special value called default value.
First test: default values for all parameters. Vary a value in each test.
Test #. Call Type Billing Access Status
1. Local Caller Residential Success
2. Long Distance Caller Residential Success
3. International Caller Residential Success
4. Local Collect Residential Success
5. Local Free Residential Success
6. Local Caller Commercial Success
7. Local Caller Dedicated Success
8. Local Caller Residential Busy
9. Local Caller Residential Blocked
 The Default Value Strategy
The default value strategy:
• determines a default value for each parameter,
• starts with a test case in which all parameters
take default values (the default test case),
• varies one parameter from its default value after
each test, and
• repeats the above step until every value of every
parameter has appeared in the tests at least once.
Systematic Single Mode Fault Detection
• Single mode fault: A fault caused by a single parameter.
Example: A switch fails whenever a free call is made.
• By design of its algorithm, the default value strategy
guarantees to detect all single mode faults.
Q: The strategy has an obvious characteristic. What is it?
Q: For the Telephone Call Problem, can we use fewer tests
but still guarantee detection of all single mode faults?
Lesson: If the only goal is to detect all single mode faults,
the default value strategy is sufficient but not necessary!
 Single vs Double Mode Faults
• Single mode fault: A fault caused by a single parameter.
Example: A switch fails whenever a free call is made.
• Double mode fault: caused by 2 parameters’ interaction.
Example: A switch fails when a collect call is blocked.
Q: Can default value strategy detect a double mode fault?
A: __________________________
Q: If not always, then how likely?
Q: Can we design a strategy to guarantee the detection of
all double mode faults? How?
 Pairwise Coverage
Q: For the telephone call problem,
(a) How many pairs of parameters are there?
(b) How many possible pairwise interactions of the parameter
values are there?
(c) How many of these interactions are tested by the default
value test plan?
A: (a) Number of parameter pairs = _____________
(b) For each parameter pair, no. of distinct value pairs = ______
So there are totally ________ distinct value pairs.
(c) In the default value test plan, only ___ value pairs are tested
for each pair of parameters.
So only ________ distinct value pairs are covered.
 Multiple Mode Fault and FTFI Number
• Multiple mode fault: A fault caused by the interaction
of the values of n parameters or conditions where n > 1.
• Failure-Triggering Fault Interaction (FTFI) number:
Number of conditions required to trigger a failure.
Q: A switch fails only whenever a free commercial long distance
call is made to a busy phone. What is the FTFI? A: FTFI = ___
Q: A microwave oven control fails only when set in reheat mode on
High power for 20 minutes. What is the FTFI? A: FTFI = ___
• To guarantee detection of n-mode faults, need to test
all n-tuples (n-way combinations) of parameter values.
• Often the FTFI is low  testing all n-way combinations
provides “pseudo-exhaustive testing”. (Kuhn et al., 2004)
 Empirical Observations on FTFI
• The percentage of faults triggered by t-way interactions increases
sharply when t = 2, then steadily to 100 as t increases towards 6.

** TCAS study used seed faults; all

others are “naturally occurring”

Source: http://csrc.nist.gov/groups/SNS/acts/ftfi.html
The Combinatorial Design Approach
• Identify parameters that determine possible
scenarios for the system under test, e.g.,
– configuration parameters
– fields on screens
– user inputs
– internal/external events
• Generate test cases that cover all pairwise, triple
or n-way combinations of the test parameters
specified in formal test requirements.
• May be used in unit, system, interoperability tests
 Example: Voice Response Unit
Consider a voice response unit with 4 parameters:
Announcement Digits wanted Billing Access type
None None Yes Local phone line
Interruptible Fixed No Long-distance trunk
Non-interruptible Variable

Constraint: The two “None” values must not be combined.

Q: How many different possible test scenarios are there?
A:
Q: How many test scenarios cover all pairs of values?
A:
 In-Parameter-Order (IPO) Algorithm
Problem: How to construct a pairwise test set?
• Create a pairwise test set for first 2 parameters.
• Extend current test set for pairwise coverage of first 3
parameters, then 4 parameters, and so on.
• Horizontal growth
– add one value (column) of the new parameter
• Vertical growth
– add new tests (rows) if necessary
• Example:
– Parameter A has values A1 and A2
– Parameter B has values B1 and B2
– Parameter C has values C1, C2 and C3
 Example: Horizontal Growth
• Fill the first two columns to
satisfy pairwise coverage # A B C
• Since C has 3 values, extend the 1 1 1
first 3 rows with C1, C2 and C3 2 1 2
• Choose C value for the 4th test to
3 2 1
cover the missing AC/BC pairs:
– Adding C1 will cover ___________ 4 2 2
– Adding C2 will cover ___________
– Adding C3 will cover ___________
• Therefore, choose __ for 4th test.
See Tai & Lei’s Algorithm IPO_H.
 Example: Vertical Growth
Q: What are the missing pairs now?
A: __________________________ # A B C
• Basic idea: 1 1 1
– Re-use existing tests if possible
– Generate new tests if needed 2 1 2
– Use wildcard “” for “don’t care” 3 2 1
• Generate ________ to cover ____ 4 2 2
• Generate ________ to cover ____
• To cover ____ , use _______
• To cover ____ , use _______
See Tai & Lei’s Algorithm IPO_V.
IPOG: Extension to t-way Coverage
• The IPO-General (IPOG) Algorithm (Lei et al., 2008)

– Construct a t-way test set for the first t parameters

– Extend the test set to cover each of the remaining
parameters one by one
• Horizontal growth - extends each existing test by adding
one value for the new parameter
• Vertical growth – adds new tests, if needed, to make the
test set complete
 IPOG Example 3-way test set
A B C A B C D A B C D
0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 1 1 0 0 1 1
0 1 0 0 1 0 2 0 1 0 2
0 1 1 Horizontal 0 1 1 0 Vertical 0 1 1 0
1 0 0 Growth 1 0 0 1 Growth 1 0 0 1
1 0 1 1 0 1 2 1 0 1 2
1 1 0 1 1 0 0 1 1 0 0
1 1 1 1 1 1 1 1 1 1 1
1 0 1 0
0 1 0 1
• Four parameters: A, B, C, D 0 0 1 2
• Each of A, B, and C has 2 values 1 1 0 2
 0 0 2
• D has 3 values  1 1 2
A Simple Application
 How Many Tests?
• There are 10 effects, each can be on or off.
• All combinations: 210 = 1,024 tests (too many!)
Q: Consider 3-way interactions only. How many?
A: ______________________________________
??? Naively: ____________ triples. We can pack 3
triples into each test. So we need about ___ tests?
Q: In reality, how many tests are actually needed?
A:
All Triples Take Only 13 Tests!
 Reading and Tool
 L. Copeland (2004). A Practitioner’s Guide to Software Test
Design. Artech House. Chapter 6.
 K.-C. Tai and Y. Lei (2002). “A test generation strategy for
pairwise testing”. IEEE Transactions on Software Engineering,
vol. 28, no. 1, pp. 109‒111.
 Y. Lei, R. Kacker, D.R. Kuhn, V. Okun & J. Lawrence (2007).
“IPOG – A general strategy for t-way software testing”. In
Proceedings of 14th IEEE Conference on Engineering of
Computer-Based Systems, pp. 549‒556.
• A list of pairwise testing tools ( http://www.pairwise.org/tools.asp )
• A tool, examples and a short tutorial ( http://testcover.com )
 Reference
• A.P. Mathur (2013). Foundations of Software Testing. Chapter 6.
• D.R. Kuhn, D.R. Wallace & A.M. Gallo (2004). “Software fault
interactions and implications for software testing”. IEEE Trans-
actions on Software Engineering, vol. 30, no. 6, pp. 418–421.
• R. Kuhn, Y. Lei & R. Kacker (2008). “Practical combinatorial
testing: Beyond pairwise”. IEEE IT Professional, vol. 10, no. 3.
• S.K. Khalsa & Y. Labiche (2014). “An orchestrated survey of
available algorithms and tools for combinatorial testing”. In
Proceedings of 25th IEEE International Symposium on Software
Reliability Engineering, pp. 323‒334.
• D.M. Cohen, S.R. Dalal, J. Parelius & G.C. Patton (1996). “The
combinatorial design approach to automatic test generation”.
IEEE Software, vol. 13, no. 5, pp. 83–88, Sep 1996.
[End of slides on Combinatorial Testing]