Critical Testing Processes

Critical Testing Processes
12 Testing Processes and Why They Matter
Rex Black Phone: +1 (830) 438-4830

RBCS, Inc. Fax: +1 (830) 438-4831
31520 Beck Road www.rexblackconsulting.com
Bulverde, TX 78163 USA rex_black@rexblackconsulting.com
Rex Black Consulting Services, Inc.

Technical and Project Management Expertise for Quality
Why Care About Testing Processes?

Bugs are expensive
US economy loses $60B yearly due to bugs, $20B
of which could be saved through better testing
Bugs drive away customers
Handwriting recognition bugs in Newton cost
Apple the person digital assistant (PDA) market
Bugs sometimes kill people
Therac-25 overdoses, Patriot clock overflow
Inadequate testing endangers the project
Bad testing processes can increase the risk of
project delay or cancellation by 25 to 300%
Critical Testing Processes Copyright (c) 2003 Rex Black Page 2
1
What is a Critical Testing Process?

Process: A sequence of actions, observations, and
decisions
Testing: Assessing the quality of a system
A test processes becomes critical when it is…
Repeated frequently → affects efficiency
Highly cooperative → affects team cohesion
Visible to peers and superiors → affects credibility
Linked to project success → affects effectiveness
In other words, critical test processes directly and
significantly affect the test team’s ability to find bugs
and reduce risks through valuable information and
services

What Are the 12 Critical Testing Processes?

1. Testing 7. Test system
2. Establishing development
context 8. Test release
3. Quality risk management
analysis 9. Test execution
4. Test estimation 10. Bug reporting
5. Test planning 11. Results reporting
6. Test team 12. Change
development management
2
The Testing Process

What is it? Why does it matter?
Planning: figuring out Reduces costs by
what testing to do finding important bugs
Preparing: building the Provides useful
tests and test team information about less
Performing: installing important bugs
the system under test Reduces risk by
and testing it identifying what works
Perfecting: reporting and what doesn’t
findings and guiding Gives management
the process essential information

A Quick Case Study in Testing ROI

So, can we agree conceptually that testing matters?
“Don’t give me that touchy-feeling [stuff],” say the
executives, “show me the money!”
Can we quantify the return DetectionCosts
on the testing investment?

Test Budget $1,000,000
Future Value of Assets Created 100,000
Regression Test Costs 500,000
Net Detection Costs $400,000
Must-Fix Bugs Found During Testing 1,500
Approximate Cost per Bug 267
Yes!
InternalFailureCosts
Pre-Release Bug Fix Costs 750,000 Assume 30 developers, burdened salary rate of $100,000 per year, three months (System Test period fixing bugs)
Regression Test Costs 500,000
Click on the thumbnail to Net Internal Failure Costs
Must-Fix Bugs Found During Testing
Approximate Cost per Bug

$1,250,000
1,500
833
walk through a case External Failure Costs

Sustaining Costs
Percentage Bug-Related
3,000,000
50%
Assume 30 developers+10 testers+20 tech support, burdened salary rate of $100,000 per year, six months after initial release.
Net External Failure Costs $1,500,000
study of testing ROI Must-Fix Bugs Found After Testing
Approximate Cost per Bug

500
3,000
from an Internet Return on Investment

Cost of Quality (w/testing)
Cost of Quality (w/o testing)
Net Benefit of Testing
3,150,000
6,000,000
2,850,000
Approximate Testing ROI 713%
appliance test project
This case study appears in my article, “ What System Development Managers Should
Know about Testing: How to Analyze the Return on the Testing Investment ,” published
by CRC Press in Systems Development Management, December 2003.
3
Testing and Constituent Processes
To optimize the overall testing process—to provide the best

possible, most valuable, most timely testing information and
services—we must optimize each constituent testing process

Establishing Context
Studying documents Aligns testing within
Talking to participants the project
Fitting in organization Aligns testing with the
Testers’ relationship to organization
the project team Clarifies expectations
Project team testing on all sides
Stakeholders
Establishes the
Fitting in operation groundwork for
System lifecycle tailoring all other
Testing deliverables testing processes
Hand-offs
4
Quality Risk Analysis

Discussing potential Identifies the key risks
problems with to system quality
stakeholders Aligns testing with the
Prioritizing the levels of key risks to system
risk based on likelihood quality
and impact of potential Builds quality and test
problem stakeholder consensus
Applying FMEA, ISO around what is to be
9126, or other risk tested (and how much)
analysis processes and what is not to be
Documenting findings tested (and why)

Test Estimation
Identifying the testing Balances costs and time
tasks, resources, and required for testing
dependencies through a against project needs
work-breakdown- and risks
structure Accurately, actionably
Drawing on test and forecasts the tasks and
project team wisdom duration of testing
Putting together a Demonstrates the
budget return on the testing
Selling the estimate to investment
management
5
Test Planning
Thinking through the Builds consensus and
strategic and tactical commitment among
details of testing test team and broader
Creating a document as project team
a communication tool participants
Reviewing and Creates a detailed map
finalizing the document for all test participants
with the test and project Captures information
team for retrospectives and
future projects

Test Team Development

Identifying the critical The testing is only as
skills needed on the test good as the team that
team does it
Hiring people who Matches test team skills
have those skills to the critical test tasks
Growing the skills of Assures competence in
the existing and new the critical skills areas
team members in line Continuously aligns
with the critical skills team capabilities with
Continuously revising organizational value of
the list of critical skills testing
6
Test System Development

Identifying specific tests Ensures coverage of the
to address critical critical risks to system
quality risks quality
Designing and Creates tests that
implementing test reproduce the
cases, data, scripts, customers’ and users’
oracles, and experiences of quality
environments Balances resource and
Documenting tests and time requirements
test coverage against criticality of risk

Test Release Management

Delivery of a known set If we ain’t got it, we
of components for can’t test it
testing If it doesn’t work in the
A hand-off from the test environment, we
project team to the test can’t test it
team If each test release is not
Balancing need for better than the one
quick feedback against before, we’re not on a
need for testing path for success
progress
7
Test Execution
Prioritizing tests with Generates information
risk about bugs, what
Assigning specific tests works, and what
to specific people doesn’t (i.e., this is
where the value of
Putting the system
testing is created)
under test into
interesting states and Consumes significant
seeing what happens resources
Gathering results Occurs at the end of the
Fine-tuning the test project and gates
project completion
work that remains

Bug Reporting
Describing a specific Creates an opportunity
way in which the to improve system (and
system doesn’t work save money)
properly Delivers (part of) the
Giving key information value of testing to the
To developers project team
To managers Builds tester credibility
To other testers with programmers
To tech support
A hand-off from the test
team to the project team
8
Results Reporting
Understanding project Provides management
goals and the questions with the information
we need to answer needed to guide the
about those goals project
Developing metrics, Delivers (another part
graphs, and charts that of) the value of testing
answer those questions to the project team
Presenting information Separates message from
and findings of testing messenger
A hand-off from the test Builds tester credibility
team to the project team with managers

Change Management
Balancing risks and Allows testing and the
costs of changes/bug project team to respond
fixes with opportunities to what they’ve learned
and benefits so far
Balancing schedule, Selects the right
budget, features, and changes in the right
quality order (i.e., focuses our
Scheduling changes and efforts on the highest
fixes for particular test ROI activities)
releases and/or system
releases
9
Okay, Rex, Now What?

Assess
How are we doing now?
Recognize challenges
Why is it so hard to do this testing stuff
right?
Improve
Where do we want our testing processes to
go?

Top 5 Assessment Questions to Ask

Is our testing effective?
Do our testers find the scary bugs, or do our customers?
Is our testing efficient?
What is the return on our testing investment?
Does testing pervade our projects?
Do testing and quality risk management start on day one?
Does testing provide the information needed, when
needed, to make smart decisions?
Have we considered test results in managing our projects
and risks?
Are our testing processes and people improving?
What data and feedback drives continuous improvement?
10
Top 5 Challenges to Good Test Processes

Unrealistic expectations
Testing can provide credible, timely information, but it does
not sprinkle magic quality pixie dust on systems
Immature development or maintenance processes
Chaos leaking into the test process makes it impossible to
deliver valuable information and services
Adversarial relationships
Testing provides information, services to entire organization
“Addicted to test”
Testers compliment—not replace—programmer unit testing
Inadequate time, resources, or involvement
Too little, too late finds few bugs, only slightly lowers risk

5 Easy Steps to Improve Your Test Processes

1. Identify gaps and problems in your current test
processes, and prioritize them
2. Plan the process improvement and sell participants
and stakeholders on your plan
3. Implement the change and measure the
improvement over time
4. Institutionalize the change so it becomes “the way
things are done”
5. Go back to step one and do it all over again!
Done right, testing is a smart investment that saves
money and reduces risk. Save more money, reduce risk
further: Optimize the twelve critical testing processes.
11

Critical Testing Processes

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Critical Testing Processes

Uploaded by

Copyright:

Available Formats

Critical Testing Processes

12 Testing Processes and Why They Matter

Rex Black Phone: +1 (830) 438-4830

Rex Black Consulting Services, Inc.

Why Care About Testing Processes?

What is a Critical Testing Process?

Rex Black Consulting Services, Inc.

What Are the 12 Critical Testing Processes?