Quality - Innovation - Vision The ROI of Testing

Presented By:
Shaun Bradshaw Questcon Technologies sbradshaw@questcon.com

Return on Investment

Slide 2

an organization can assess whether the expense/investment is justified by the resulting savings/revenue Slide 3 .Ret r rn n I vest ent (ROI) Invest What is ROI? ROI is a calculation that attempts to determine the actual or perceived future value of an expense or investment By calculating ROI.

Ret rn n Invest ent (ROI) How is ROI calculated? In its most basic form ROI is calculated as follows: ROI = (Benefits ± Costs)/Costs That is. calculated as a percentage of those costs Benefits Revenue generated Cost reduction Cost avoidance Productivity improvements Slide 4 Costs Labor expense Time expense Tool expense . the financial benefit after an investment or improvement is made minus the cost of the investment or improvement.

Ret rn n Invest ent (ROI) What makes testing valuable? How do we get to a positive ROI? Produces information reliably grounded in observed system behavior [1] Functions like a credit check [1] Exposes risk and improves decision-making Extends the life of the software Improves the development process Increases end-user confidence and satisfaction Slide 5 .

Defect Detection Slide 6 .Defect Prevention vs.

Defect Detection Importance of Testing Organizations rely on testers to do QA We cannot test quality into an application Test needs to be the backdoor to QA Requires a shift in the testing paradigm Test Finding Defects Measuring Quality Slide 7 .Defect Pre ention s.

Defect Detection Rules & Results of Defect Prevention 60% Rule [2] Percentage of defects introduced in requirements and design 85% Rule [2] Percentage of defects removed through reviews.Defect Pre ention s. inspections and testing 368:1 Rule [3] Slide 8 .

100 $378.600 368X+ 0 15 15 $212.700 .Defect Pre ention s. Defect Detection Test Only Costs Test Only Costs Test 300000 300000 250000 250000 200000 200000 150000 150000 100000 100000 50000 50000 0 Testing Only Re Re q u Re quiire qu re me ir me nt Re e m nts s qu e nt s ire m en De ts sig De n sig Re n qu ire Dm ee Cs oind gtn e s Assume X is $100 & 100 total defects X # of defects introduced Accumulated defects Defects remo ed Cost Accumulated Cost Slide 9 30 30 0 $0 $0 Requirements Design Code e Te De s sig t n Pr Co od de T uc e tiost n Co d Test Production 5X 30 60 0 $0 $0 10X 30 90 0 $0 $0 50X 10 100 85 $166.600 $166.

Slide 10 .

Quantitative Benefits Slide 11 .

000 $200.000 $0 o A H e tn n e tO QA Fu Slide 12 .000 $100.Quantitati e Benefits Reduced Defect Repair Costs Test involvement earlier in the lifecycle reduces repair costs Automation improves defect detection Good test practices increase detection likelihood $600.000 $500.000 $300.000 $400.

the Annuity Nightmare Small rounding error goes undetected at a major financial institution Months until discovered Code repair is cheap Production cost NOT CHEAP! Increased Revenue & Profits Reduced time-to-market for functioning software Increased market share Higher customer retention/goodwill Reduced maintenance costs (easier to enhance) Slide 13 .Quantitati e Benefits Decreased Production Costs Defects in production can be significantly higher than 368x .

Calculating Test Value Slide 14 .

Key Metrics to Calculate Test Value Defect Injection Rate Total number of defects introduced into an application When each defect was introduced (using Root Cause Analysis) Number of defects introduced in each phase of the SDLC Calculate the defect injection rate as: # of defects introduced total defects Slide 15 .

) Defect repair time for each resource by phase of the SDLC Number of defects repaired in each phase of the SDLC Calculate the defect repair costs as shown in the example*: *Data in the example was drawn from Slide 16 [4]. . BA. Developer. etc. Tester.Key Metrics to Calculate Test Value Defect Repair Cost Hourly rate of each resource involved in the repair (PM.

Key Metrics to Calculate Test Value Test Effectiveness Number of defects found during testing activities during any phase of the SDLC Total number of defects found including in production (use a warranty period ) Calculate test effectiveness as: # of defects foundphase total # of defects found Slide 17 .

ROI Example Slide 18 .

ROI Example Slide 19 .

500 Slide 20 .ROI Case Study Established a test team made up of: 1 test lead 1 test analyst 1 test consultant Total 1st year investment: $237.

ROI Case Study Conducted 2 projects in first 8 months Realized $173.000 savings in project-over-project defect repair costs Calculated savings do not include other benefits such as: Reduced production downtime Reduced maintenance costs Increased customer satisfaction & trust Anticipated 46% ROI at the end of year 1 Slide 21 .

com 336-273-2428 .Questions Slide 22 sbradshaw@questcon.

jsp> US Dept. Planning Report 02-3: The Economic Impacts of Inadequate Infrastructure for Software Testing. WV: NASA IV&V Facility.html> Dabney. "Return on Investment of Independent Verification and Validation Study Preliminary Phase 2B Report. Bullock. Calculating the Value of Testing. <www. May.mil/crosstalk/2005/04/0504Jones.gov/ViewResearch/289/24. <www. 2005. 3. James. National Institute of Standards & Technology (NIST). <www. <sarpresults.hill. Slide 23 . Software Testing & Quality Engineering: May/June. "Software Cost Estimating Methods for Large Projects. 2002. JB.stickyminds. 2003.pdf> 2. Strategic Planning & Economic Analysis Group.af.gov/director/prog-ofc/report02-3. of Commerce. Technology Program Office. Capers.References 1. 2000.stsc." Fairmont." CrossTalk: April.nasa.nist. 4.com/> Jones.ivv.

Sign up to vote on this title
UsefulNot useful