IBM

Effective Test Coverage

Techniques
Functional test coverage
Shruthi V Gowdru & Saravana Kumar KKB

9/4/2014

About the authors: Shruthi,

working as a Software Engineer
(QA) for IBM Policy Atlas team
under ECM. She has 4 year’s
experience in QA. Reach out to
her at shgowdru@in.ibm.com
Saravana, working as a QA lead
for IBM Policy Atlas team under
ECM. He has total 10+ years’
experience in QA. Reach out to
him at saravkkb@in.ibm.com

Abstract: One of the major challenges in Testing is whether all possible scenarios and possible
cases are covered for the mentioned requirement. There is no standard answer for this and it is
difficult to say “Yes” it is covered. If it is then how can we ensure it is covered? This paper is
about one of the test case designing technique, on coming with various possible cases by deriving
the Factors and corresponding values.
Effective Test Coverage Techniques

1 Introduction:
The quality and effectiveness of software testing is largely determined by the quality of
the test cases we develop. The challenge of providing coverage through the test plan is
very important, for full functionality coverage. However, in Software Testing there is no
quantitative method to measure the Test coverage.
This paper will describe a methodology of coming up with Test Coverage Matrix using
“Factor/Value” concept. The same has been described in detail in below sections.

2 Deriving Factors and Values:

Factors are the Major entities that specify the functionality of the product and all possible
combinations that a factor can have, specifies the Values in the test plan.

Let’s see the example of how we can develop the test cases using the Factors/Values.

Example: Requirement: New installer support for a sample application.

Below are the requirements specified:

 Installer should support fresh and upgrading from older version of application
 Uninstallation of application should be suppored and it should be done cleanly

II. Testing factors that affect the outcome of a run:

Identify testing factors which affect the outcome of a scenario execution.
Factor Description
Factor1
Factor2
Factor3
Factor4

Example:

1. Testing factors that affect the outcome of a run:

Factor Description
Factor1 INSTALLATION
Factor2 UNINSTALLATION

2
Effective Test Coverage Techniques

III. Values of each factor:

Identify the Values of Each Factor which cause different outcomes.
Factor Value Description
Factor1-Value1
Factor1 Factor1-Value2
Factor1-Value3
Factor2-Value1
Factor2-Value2
Factor2
Factor2-Value3
Factor2-Value4
Factor3-Value1
Factor3
Factor3-Value2
Factor4-Value1
Factor4 Factor4-Value2
Factor4-Value3

Example:
Factor Value Description
Fresh
INSTALLATION Upgrade from older
version
From control panel/
uninstall script
UNINSTALLATION
Double clicking
installer file

3 Analysis of the factors and values:

Divide the testing space into mutually exclusive sections here. Show reasoning why not all
permutations or combinations of the factors values need to be tested. Pick out the most important
factors and their combinations should always be fully covered. A/B/C and 1/2/3/4 are the
coordinate which can indicate one combination. Below are the sample template and example of
factors and its values.

IV. Permutations/combinations analysis of the factors values

NOTE: X means should be covered; N/A means not reachable combination; Ignored means
meaningless combination. You can change the style of the following table if you have more than
two key testing factors.

3
Effective Test Coverage Techniques

Key Combination Table

Factor2- Factor2- Factor2- Factor2-
Values Value1 Value2 Value3 Value4
1 2 3 4
Factor1-Value1 A X X X X
Factor1-Value2 B X N/A X X
Factor1-Value3 C X X Ignored X

Example:
Uninstalling
Uninstalling
From control
Double clicking
Values panel/ uninstall
installer file
script
1 2
Fresh
A X X
Installation
Upgrade
B X X
Installation

4 Test Coverage Matrix:

Test Coverage Matrix is a mapping tool to map the actual test cases with the derived test
conditions. As mentioned above it the lowest level of breakdown structure as it points to the actual
test case itself. Below is the sample template and example of Test Coverage Matrix.

V. Test Coverage Matrices:

ORANGE stands for the coverage of key combinations; GREEN stands for common coverage.
“A1” means key combination A1 has been covered by Test Scenario 0005.

Table A. Factor1: Factor1-Value1

Here Factor1 will be used to divide test scenarios into different matrices. Use A/B/C as the table
ID; use Factor1 and its value as the title of the table.
Factor Value 0001 0002 0003 0004 0005 0006
Factor1 Factor1-Value1 X X X X X X
Factor2-Value1 A1
Factor2-Value2 A2
Factor2
Factor2-Value3 A3 A3 A3
Factor2-Value4 A4
Factor3-Value1 X X X
Factor3
Factor3-Value2 X X
Factor4-Value1 X X
Factor4 Factor4-Value2 X X X
Factor4-Value3 X X

4
Effective Test Coverage Techniques

Table B. Factor1: Factor1-Value2

Factor Value 0007 0008 0009 0010
Factor1 Factor1-Value2 X X X X
Factor2-Value1 B1 B1
Factor2-Value2
Factor2
Factor2-Value3 B3 B3
Factor2-Value4 B4
Factor3-Value1 X X X
Factor3
Factor3-Value2 X
Factor4-Value1 X X X
Factor4 Factor4-Value2
Factor4-Value3 X X

Example:
Factor Value 0001 0002 0003 0004
Fresh Installation X X
Installation Upgrade
X X
Installation
From control
A1 B1
panel
Uninstallation Uninstalling
Double clicking A2 B2
installer file

The Abstract for Each Scenario:

Describe each test scenario designed for this requirement in one line.

Test
One Line Description TC ID
Scenario ID

Example:
Test
One Line Description TC ID
Scenario ID
Verification of fresh installation and uninstallation from
0001
control panel
Verification of fresh installation and uninstallation by double
0002
clicking the installer file
Verification of Upgrade installation and uninstallation from
0003
control panel
Verification of Upgrade installation and uninstallation by
0004
double clicking the installer file

5
Effective Test Coverage Techniques

5 Benefits:
1. This approach enables test team to cover the breadth oriented testing
2. The completeness of test coverage matrix reduces the gaps in the test cases
3. Premutations and combinations of the factors and vlaues helps in largesly uncover all the
scenarios
4. Overall all these Benefits give a better quality product with reduced in number of defects can
occur in the field.