2011 27th IEEE International Conference on Software Maintenance (ICSM)

3UDFWLFDO&RPELQDWRULDOWZD\0HWKRGVIRU'HWHFWLQJ
&RPSOH[)DXOWVLQ5HJUHVVLRQ7HVWLQJ
Rick Kuhn (kuhn@nist.gov) and Raghu Kacker (raghu.kacker@nist.gov)
National Institute of Standards and Technology
Gaithersburg, MD 20899
Abstract— Regression testing can be among the most
challenging of software assurance tasks because program
changes often introduce faults, including unexpected
interactions among different parts of the code. Unanticipated
interactions may also occur when software is modified for a
new platform. Techniques such as pairwise testing are not
sufficient for detecting these faults, because empirical
evidence shows that some errors are triggered only by the
interaction of three, four, or more parameters. However, new
algorithms and tools make it possible to generate tests that
cover complex combinations of values (2-way to 6-way), or to
analyze existing test suites and automatically generate tests
that provide combinatorial coverage. The key advantage of
this approach is that it produces better testing using a fraction
of the tests required by other methods.
Keywords-combinatorial testing; regression testing; coverage;
I. INTRODUCTION
Developers of large data-intensive software often notice an
interesting—though not surprising—phenomenon: after a
minor change, components that have operated for months
without trouble suddenly develop previously undetected
errors. For example, the application may have been installed
on a different OS-hardware-DBMS-networking platform, or
newly added features fail on an oddball combination of values
that have not occurred before. Some of these rare
combinations trigger failures that have escaped extensive
regression testing. Such failures are known as interaction
failures, because they are only exposed when two or more
input values interact to cause the program to reach an incorrect
result. Recognizing that system failures can result from the
interaction of conditions that might be innocuous individually,
software developers have long used “pairwise testing”, in
which all possible pairs of parameter values are covered by at
least one test. But what about the remaining faults? How
many failures will be triggered only by an unusual
combinatorial interaction of more than two parameters?
A study of 15 years of medical device software failures
found one case in which a failure involved a four-way
interaction between parameter values. Subsequent
investigations found a similar distribution of failure-triggering
conditions: usually, many were caused by a single parameter
value, a smaller proportion resulted from an interaction
between two parameter values, and progressively fewer were
triggered by 3, 4, 5, and 6-way interactions (Figure 1), a
relationship we refer to as the interaction rule.
U.S. Government work not protected by U.S. copyright
599
These results suggest that combinatorial testing which
exercises a high degree (4-way or above) of combinatorial
interactions is necessary to reach a higher level of software
assurance. If we know from experience that t or fewer
variables are involved in failures for a particular application
type and we can test all t-way combinations of discrete
variable settings, then we can have reasonably high confidence
that the application will function correctly.
Figure 1. Percentage failures triggered by t-way interactions
II. HOW DOES COMBINATORIAL TESTING WORK?
The key ingredient for this form of testing is known as a
covering array, a mathematical object in which all t-way
combinations of parameter values are covered at least once.
Generating covering arrays for complex interactions (beyond
pairwise) is a difficult problem, but new algorithms have been
developed that make it possible to generate covering arrays
several orders of magnitude faster than previous algorithms.
These algorithms have been incorporated into practical tools
that are used by hundreds of organizations.
Covering arrays can be used in initial unit or system testing,
but new methods and tools have been developed to analyze an
existing test suite, produce detailed measurements of its
combinatorial coverage, and automatically extend it to achieve
any desired combinatorial coverage up to 100%. These tools
make combinatorial testing significantly easier to accomplish
in regression testing, or to integrate this advanced testing
approach into a traditional testing paradigm.
Combinatorial testing has spread rapidly because it can
produce better testing at lower cost. More than 700
organizations use NIST’s tools, and others, such as Microsoft
PICT, have similarly large user bases. This tutorial seeks to
increase awareness of this effective test approach, and equip
regression testers with knowledge and tools to apply it to
testing and assurance in software maintenance.