You are on page 1of 19

Learnings of the 8 th

Annual Software testing
•Regression Test Optimization
•eXtreme Testing
•Test Effectiveness

Sanjay Kompally
UAT - Workstation
Test Effectiveness
What is Test Effectiveness?
Is it the rigor of testing?
Is it the speed of testing?
Is it the ROI (or Value) of testing activity?
Is it the improvement in quality of product before &
after Testing
Or does Test Effectiveness mean all of these?

GE Title or job number /
Rigor of Testing

Requirement Coverage
Risk Coverage
Rounds of Testing
Competency of Testers

GE Title or job number /
Test Effectiveness
Requirements Coverage
Requirement-to-Test traceability matrix is an important technique in
assuring adequate coverage.

Optimization techniques like orthogonal array technique help in
number of test cases without compromising on coverage or defect
detection potential by eliminating redundancy which is otherwise
not very

Risk Coverage :
While requirements coverage primarily re-assure that what was
is actually working fine, it is the risk based testing which is closer to
the desired “destructive” approach.

Risk analysis or potential failure possibility identification is a very
important defect detection/prevention perspective. 4/
GE Title or job number /
Rigor of Testing
Rounds of Testing

Rounds of testing are dependent on when to stop testing
criteria. It might be easier to estimate number of rounds of
testing if the criteria are based on time duration or number
of test cases to be executed.

Iterative testing (which implies more rounds of test) has
benefits. Results of the previous rounds are used to guide
the successive rounds of testing. The feedback could be
used in many ways:
· Increasing test representation for poorly covered areas
· Increasing test representation for highly buggy areas

GE Title or job number /
Rigor of Testing
Tester Competency

In spite of best test strategy, excellent tools and grand test
budgets, test
effectiveness is only as good as the competency of testers.

Testers are the Sherlock Holmes of the s/w

GE Title or job number /
Speed of Testing
Speed of testing is less about how many test cases can be
executed per unit time, but more about how many defects can
be detected per unit test effort

GE Title or job number /
Defects in Testing

GE Title or job number /
eXtreme Testing
Extreme testing (XT) derives from the eXtreme Programming (XP) & Distributed
Programming(DXP) agile development methodologies and implements its core
practices like
Planning Game, Small Releases, Metaphor, Simple Design, Pair Programming,
Ownership, Continuous Integration, 40-hour Week, On-Site Customer.

The main focus of this is –
· Quick and lean response to changing customer requirements from a QA
· How extreme testing (XT) can be added as a plug-in to a conventional
. This paper attempts to outline a methodology to make the QA process truly

This approach retains the classical concepts of eXtreme Programming like
simplicity, discipline, communication, quality,and flexibility.

GE Title or job number /
3.0 Preliminaries
The following section outlines how most of the 12 practices of DXP can be
applied to conventional software testing methodology namely,
· Planning Game
· Small Releases
· Metaphor
· Simple Design
· Testing,
· Refactoring
· Pair Programming
· Collective Ownership
· Continuous Integration
· 40-hour Week
· On-Site Customer
· Coding Standards
The principals from above will be adopted into extreme testing on a strict need
to have basis and should be aggressively tailored to appreciate the fact that the
process was initially designed for software development by keeping testing as a
back end activity.
10 /
GE Title or job number /
Planning Game
The “planning game” concept address the following key decisions:
· What will be accomplished by the due date?
· What to do next?
The extreme testing (XT) approach will also answer these 2 questions.
The tailored form of it being:
· What deliverables will be tested by the due date?
· What deliverables will be accepted into QA?

It works in iterations of 2 weeks. Where the planning game is re-
envisioned and re-factored at the end of every 2 weeks

11 /
GE Title or job number /
The core of this activity lies in the fact that
·All modules that can be tested in parallel SHALL NOT be accepted into
QA even if a build is available for deployment
·The focus of eXtreme Testing (XT) is to aggressively test a select list of
modules in a burst and NOT lukewarm testing of multiple modules

12 /
GE Title or job number /
Advantages of this model of planning

· The number of defects reported at any given point will cluster around
a select list of modules.
· The tracking of bugs is less tedious. Phase containment of to a micro
· The development team can focus on the clean-up of a restricted
number of modules.
· The modules that will be tested are prioritized by the customer.
· The customer perceived value addition in terms of a bug free roll-out
of his high priority modules will be enhanced
· The completeness of testing is inherently transparent to the customer
since the team is entirely focused on a select list of high priority

13 /
GE Title or job number /
The concept of pair testing can be adopted when the team is extremely focused on
testing a very few modules in parallel, that too extremely aggressively. The test pair
formed should then resort to classical testing concepts like “exploratory testing” and
“intuitive testing” since 2 resources are deployed for testing 1 module the productivity of
bugs reported to testing efforts should not be
The benefits of pair testing are:
· Early reporting of a large number of defects.
· Enhance the customer perception that the test team is highly efficient in spotting

This is an approach where the testing team evolves a common and complete
understanding of
what the system is intended to do at a macro level and what each piece of it coming into
during a 2 week burst is supposed to address. The implementation of this concept is done
through brain storming sessions prior to starting a 2 week testing burst where the team
will review
and strategize the activities planned for the week to arrive at a common understanding of
modules coming in and how it integrates to the already certified modules. This can help
test 14 /
GE Title or job number /
teams evolve their understanding of the system and will help at advanced testing stages 03/12/08
like endto-
40-Hour Week
This is a mandate of eXtreme Programming (XP) that is retained as
is in eXtreme Testing (XT).
Since teams adopting eXtreme Testing would endure long periods of
concentrated efforts it is
mandatory that they are not to work overtime. It is deemed in this
paper that the necessity of
adhering to this and the impact of not adhering to it need not be
“XP teams are in it to win, not to die”

15 /
GE Title or job number /
Strategies in Regression Test
Objectives of Regression Testing
To ensure that the current system will work as per intended when
updates/modifications done to the existing system.
5 To create an approach among functional areas and information system
where continual maintenance and improvements to regression testing
become part of a normal activity.
6 To develop an optimized test suite which could be used to monitor and
maintain regression testing efforts.

One of the easiest solutions to retain the sanity of existing features is to
everything on all the product releases. However, this ensures high
quality, but the
product will have bigger cycle time and the product might be out of
market due to
delays in delivery and due to enhanced competition. Different Test
strategies 16 /
GE Title or job number /
employed during Regression Testing can help in arriving at Optimum 03/12/08
3. Strategies in Regression Test Optimization
3.1.Traditional Approach of Regression Test selection
The traditional & simple way of regression test strategy is considering complete
test suite for retesting, in which all the test cases are selected for all possible
product releases. Although this ensures high quality, this involves redundant &
huge effort and increases cycle time for delivery. Also, this approach does not
create the required value for cost spent as discovering new bugs in legacy
features are minimal.
3.2.Scientific Approach for Regression Test Selection
Nowadays, Customer expects results from every dollar spent for test execution.
So, testing should employ scientific approaches to arrive at an optimized
regression suite. Having optimized Regression test suite will help in having the
product delivered quickly and also it helps in Cost reduction. There are different
strategies in optimizing the Regression Test suite. To substantiate the details,
we take an Element Management System (EMS) which supports multiple
DSLAMs (DSL Access Multiplexers) of different Software releases. Most of these
concepts can be applied to other domains also. Organizing the regression tests
in a controlled fashion is the first step for an effective Test Strategy. The
following sections detail some of the techniques for classification of domains
and test cases.

17 /
GE Title or job number /
3.3.Doman wise classification of Test Cases
The first step towards classification of Test cases is to make every test case
available in a single place or in a central location, which should be in the form
of a Master Test List for any testing. Based on the architecture of the System
under Test and the DSLAMs used, test cases should be mapped to respective
domains. The complete Baseline regression tests should be mapped to
corresponding domains. The complete list of test cases should have
traceability to the respective domains. The table below illustrates the number
of test cases mapped to different domains for an Element Management
System (EMS) project.

Prioritization of Test Cases................................................................5
4.4.2 Prioritization of Domains...................................................................5
4.4.3 Domain Prioritization Based on Requirements......................................5
4.4.4 Domain Prioritization based on Field Bugs:........

18 /
GE Title or job number /
Regression Test Optimization
4. Scientific Approach for Regression Test Selection:
Optimized Regression Test Suite is the synonym for the current day Regression
Testing. The current approach under discussion uses the Domain priorities and Test
Case priorities for Regression Test Selection & Optimization. This approach uses the
following for arriving at Optimized Regression Suite.
22 Statistical Data for arriving at priority of different domains.
23 Test Cases Priority for selecting optimum number of test cases
24 Priority of the domains based on new requirement additions in the latest
Software release.
A Boolean “OR” is performed on the domains based on different priorities and test
case selection is planned accordingly. The following Case Study on an EMS Testing
project illustrates the approach followed for optimized Regression Test strategy using
the techniques discussed in the earlier sections.

6. Conclusion
The concepts discussed have been applied in an EMS project at Wipro for a European
Customer. This project was very critical to Wipro and it required huge effort to
ensure that the regression tests are optimum and the release cycles are short.
The prioritization of Domains and Complexity Analysis requires good Technical
Domain strength and overall product understanding. Otherwise, this exercise would
not yield the expected results.

19 /
GE Title or job number /