Defect Management & Metrics: Society For Software Quality April 24, 2010

Defect Management &
Metrics
Society for Software Quality
April 24, 2010
2010 Holl
Consulting
021209.01 SSQ Presentation
!o "s #!is $uy%
&
30 years in software
and hardware/software
Quality
&
19 years in Quality &
Process Management
&
6 years as Director of
Software Best Practices
and Corporate Software
Metrics Leader at Intuit
&
Past Q President
Q
U
A
LI
Y
T
Q S
S
2010 Holl
Consulting
'ri(ary )ele*ant +,perience
&
-esigne. "ntuit/s corporate .efect
(anage(ent process
&
Assiste. in rollout to 01100 engineers
&
$ui.e. its e*olution for si, years
&
As Corporate Software 2etrics 3ea.er,
fostere. 4uantitati*e pro5ect (anage(ent
an. process i(pro*e(ent
&
2a.e (ista6es an. learne. fro( t!e(
2010 Holl
Consulting
Agen.a
&
Su((ary of 're*ious #al6
&
-efect 2anage(ent 2etrics
&
2etrics for 'ro5ect 2anage(ent
&
2etrics for 'rocess "(pro*e(ent
&
A.*ance. 2etric 2o.els
2010 Holl
Consulting
3ast ti(e7
&
e re*iewe. cultural issues relate. to
.efect (anage(ent
&
e went o*er a .efect (anage(ent
process8
'olicies
or6flow9States
:iel.s
2010 Holl
Consulting
-ata Collection
&
At .ifferent states in t!e wor6flow, .ifferent
.ata are collecte.
&
2ost .ata are collecte. at su;(it ti(e
&
A..itional .ata are entere. ;y t!e
.e*eloper upon resolution
&
An. finally ;y t!e tester upon *erification
2010 Holl
Consulting
Agen.a

&
&
&
&
2010 Holl
Consulting
!en -oes A -efect Count%
It de!ends on your goals
&
Defect Management8 #o trac6 .efects for resolution in
or.er to release pro.ucts
<ust log functional pro;le(s foun. in test= .e*iations fro( t!e

re4uire(ents t!at t!e custo(er will see= .efects t!at you .on/t
want to fall t!roug! t!e crac6s.
&
Defect "nalysis# #o i(pro*e t!e process or pre.ict
"f your goal is to get ;etter >learn? or pre.ict t!e future, you nee.
to capture (uc! (ore infor(ation.
$ithout early lifecycle data %re&uirements' etc() !redicti*e
models aren+t that relia,le
$orld-class com!anies don+t .ust manage defects / they analy0e information(
2010 Holl
Consulting
-ata Collection Costs
& 1stimate "ssum!tions8
11,000 .efects
@ t!e .efects are correcte. >9,000?
-ata collection ti(e8 10 (inutes >*ali.ate. at H'?
A*erage .efect cost8 1A !ours >.ata fro( 2ic!ael :agan?
-ata collection re.uces .efects BC
#i(e to recor. >116 D10(? E,000 !rs
#i(e to correct >96 D1A!? 144,000
!rs
-efect re.uction >96 D
BC?
4B0 .efects
#i(e sa*ings >4B0 D 1A!? F,200 !rs
23I 4506
Sa(pling is *ali., ;ut
sa(pling !as to ;e
(anage. to get goo.
ran.o( sa(ples an. to
ensure t!at a.e4uate
>statistically significant?
sa(ples are ;eing
collecte..
#!is .oesn/t 4uantify t!e *alue of ;eing a;le to pre.ict 4uality an. sc!e.ules
2010 Holl
Consulting
Si(ple 2easures
&
-efect cost G

&
:i, rate G
&
-uration G
3ast two inclu.e 4ueue ti(e, so t!ey .epen. on wor6loa.
"t/s !elpful to un.erstan. t!e stan.ar. .e*iation too
Honus8 Hy type, root cause, in5ection p!ase
D +,cel Ietor6-ays is a useful function
J >#otal )esolution +ffort?
Iu(;er of resol*e. .efects
'erio. of ti(e >wee6, (ont!, etc.?
J>>Close. -ate? >Su;(it -ate??D
2010 Holl
Consulting
)elati*e -efect Costs
&
"f you recor. in5ection an. .isco*ery
p!ases an. effort, you can co(pute
relati*e costs
&
+,a(ple8
Cost of a re4uire(ents .efect G 1
Cost of a .esign .efect G 10
Cost of a co.ing .efect G 100
Cost of a pro.uction .efect G 1000

2010 Holl
Consulting
Si(ilar to '!ase Contain(ent
&
Start wit! (atri, of in5ection an. .isco*ery
&
+nter t!e a*erage rewor6 >or .uration?
costs in eac! cell >resol*e. .efects only?
In.ection Phase
Disco*ery Phase )e4Ks -esign Co.e #est 'ro.uction
)e4uire(ents 2.4
-esign 1B.E 4.1
Co.e 22.F 9.B 1.2
#esting 4E.1 11.B 4.E 1.2
'ro.uction 1B.E E1.B 11.E 2.2 1
2010 Holl
Consulting
Si(ilar to '!ase Contain(ent
&
Ior(aliLe ;y .i*i.ing eac! cell ;y t!e
"n5ectionG-isco*ery *alue
In.ection Phase
)e4uire(ents 1.00
-esign A.E1 1.00
Co.e 9.4A 2.E2 1.00
#esting 11.2B 4.B1 E.B1 1.00
'ro.uction EB.B4 9.E9 1B.2B 1.1E 1.00
2010 Holl
Consulting
3ogging -efects
&
#!is is not
How many defects leak?
&
#!is is
What percent of known defects are recorded?
&
"f you only start logging .efects in test, you/re
(issing o*er !alf of your .ata
&
#ry to capture .ata fro( re*iews9inspections an.
.e*eloper unit tests
2010 Holl
Consulting
'rioritiLing -efects
&
#!is is not a;out !ow to set priority, ;ut7
&
'riority (eans t!e (ost i(portant .efects get
resources first
&
#o (easure t!is, we nee. to 6now t!e 4ueue
ti(e >fro( Msu;(itN to Mwor6 startsN?
&
#!erefore, we nee. an MAccepte.N state, or
so(et!ing to note w!en wor6 ;egan
&
!y can/t we use u,mit to 7erify .uration%
2010 Holl
Consulting
Aging -efects
&
#i(e in )esol*er 4ueue
&
#i(e in Oerifier 4ueue
&
Alternately, use
.urations (inus
effort as a pro,y
"*erage Queue 8ime
0
2
4
A
1
10
12
0P#est
Stopper
1 2 E 4 B
Priority
D
a
y
s

i
n

Q
u
e
u
e
2010 Holl
Consulting
-efect Hac6log
&
2any groups .on/t count .efects left o*er fro(
prior releases= t!ey only focus on .efects foun.
an. not fi,e. in t!e current pro5ect
&
#!is is .angerous ;ecause a pro.uct .efect
;ac6log can grow un(anage.
&
ALL of t!e open .efects are release., not 5ust
t!e new ones
&
!en pro.ucing t!e report of open .efects,
4uery for all open .efects in t!e product
2010 Holl
Consulting
#!e Calculation
$ee9
1
2
E
4
8otal num,er of defects
/ :um,er closed ,efore the wee9 started
/ :um,er that were o!ened after the wee9 ended
; :um,er o!en in the wee9
2010 Holl
Consulting
Calculating Hac6log in +,cel
&
Query Su;(it -ate an. Close. -ate
2010 Holl
Consulting
Calculating Hac6log in +,cel
8his uses named ranges' where
-.Iu(.C)s G #otal nu(;er of C)s opene.
-.#.Close. G )ange of Close. -ates
-.Su;(it.-ate G )ange of Su;(it -ates
Array for(ula. QS!iftPCtrlP)eturnR
2010 Holl
Consulting
#!e )esult
Product Defect <ac9log
0
200
400
A00
100
1000
1200
1
9
F
9
2
0
0
A
2
9
F
9
2
0
0
A
E
9
F
9
2
0
0
A
4
9
F
9
2
0
0
A
B
9
F
9
2
0
0
A
A
9
F
9
2
0
0
A
F
9
F
9
2
0
0
A
1
9
F
9
2
0
0
A
9
9
F
9
2
0
0
A
1
0
9
F
9
2
0
0
A
1
1
9
F
9
2
0
0
A
1
2
9
F
9
2
0
0
A
1
9
F
9
2
0
0
F
2
9
F
9
2
0
0
F
E
9
F
9
2
0
0
F
4
9
F
9
2
0
0
F
B
9
F
9
2
0
0
F
A
9
F
9
2
0
0
F
F
9
F
9
2
0
0
F
1
9
F
9
2
0
0
F
9
9
F
9
2
0
0
F
1
0
9
F
9
2
0
0
F
1
1
9
F
9
2
0
0
F
1
2
9
F
9
2
0
0
F
1
9
F
9
2
0
0
1
2
9
F
9
2
0
0
1
E
9
F
9
2
0
0
1
4
9
F
9
2
0
0
1
=

3
!
e
n

D
e
f
e
c
t
s
2010 Holl
Consulting
Agen.a

&
&
&
2010 Holl
Consulting
So(e 'ro5ect 2anage(ent -efect 2etrics
&
)e*iews S "nspections
&
-efect -isco*ery S Closure
&
-efect -ensity
2010 Holl
Consulting
A 'ro5ect A,io(
&
#!e 4uality of pro.uct in one
p!ase is an in.icator of t!e
4uality in later p!ases
&
!y%
#!ere are two answers

2010 Holl
Consulting
)e*iew 2etrics
&
)e*iews S inspections !a*e
pro*en to ;e t!e (ost costP
effecti*e way to .etect an.
re(o*e .efects
&
"nspect your 6ey pro5ect artifacts
>re4uire(ents, .esigns, co.e, tests, etc.?
2010 Holl
Consulting
)e*iew9"nspection -ata
&
#!ere are (any .ata collecte. .uring a
re*iew or inspection
&
#!e ones t!at tell us a;out 4uality are8
Iu(;er of 3!erational .efects .isco*ere.
Iu(;er of Minor .efects foun.

per c!ange. line of inspecte. .oc9co.e
&
Iu(;er of re*iews per .oc9(o.ule
2010 Holl
Consulting
'ro5ect -efect -ensity
&
Consistency in categoriLing inspection
.efects is i(portant
Ha*e clear gui.elines for Tperational an.

2inor .efects
& -efect -ensity
'ro5ect
G
Tperational -efects foun. ;y inspection
-elta S3TC >or 3T#?
2010 Holl
Consulting
'ro5ecting -efect 3ea6age
&
"nspecte. total of BE40 S3TC
&
:oun. 4F operational .efects
&
-ensity G 4F9BE40 G 1.1 .efects9US3TC
&
'ro5ect c!ange. 11,4AE S3TC
&
+sti(ate. .efects G 11.4AE D 1.1 G 1A2
&
)e(aining G 1A2 P 4F G 11B
!at s!oul. you .o% "nspect (ore% Tr #est for t!e(%
2010 Holl
Consulting
Oerification
&
Vse -efect -ensity to esti(ate e,pecte.
yiel.
Si(ilar to esti(ate .escri;e. wit! re*iews

&
-efect .isco*ery S closure
&
Vse e,trapolation to pre.ict w!en .efect
e,it criteria will ;e (et
2010 Holl
Consulting
-efect -isco*ery
#i(e
C
u
(
u
l
a
t
i
*
e

-
e
f
e
c
t
s

:
o
u
n
.
8otal Defects 3!ened
"
.
e
a
l

C
u
r
*
e
W
$ee9ends
2010 Holl
Consulting
-efect -isco*ery Cur*e Assu(ptions
&
#i(e is a pro,y for Me,posureN
)e(o*e wee6en.s to s(oot! t!e cur*e

&
Iu(;er of test cases run woul. ;e ;etter,
assu(ing
#est co*erage is goo.
Iot continuing to a.. new test case

2010 Holl
Consulting
ellPHe!a*e. Cur*e
#i(e
C
u
(
u
l
a
t
i
*
e

-
e
f
e
c
t
s

:
o
u
n
.
)elease #arget -ate
2010 Holl
Consulting
A..8 -efect Closure
#i(e
C
u
(
u
l
a
t
i
*
e

-
e
f
e
c
t
s
8otal Defects 3!ened & >losed
Tpene.
Close.
)e(aining
Tpen
2010 Holl
Consulting
A..8 #o QA
#i(e
C
u
(
u
l
a
t
i
*
e

-
e
f
e
c
t
s
8otal Defects 3!ened' >losed' & 8o Q"
Tpene.
#o QA
Close.
Tpene., waiting for
.e*elop(ent fi,
#urne. o*er fro(
.e*elop(ent to QA,
awaiting *erification
2010 Holl
Consulting
:in.ing #!e Halance
#i(e
C
u
(
u
l
a
t
i
*
e

-
e
f
e
c
t
s
8otal Defects 3!ened' >losed' & 8o Q"
Tpene.
#o QA
Close.
$hat+s
ha!!ening
here?
QA is a
;ottlenec6
-e* is a
;ottlenec6
2010 Holl
Consulting
A..8 3egacy S #arget )elease
#i(e
C
u
(
u
l
a
t
i
*
e

-
e
f
e
c
t
s
8otal Defects 3!ened & >losed
#otal
#argete.
#o QA
Close.
"n!erite. fro(
pre*ious !istory,
targete. for or foun.
in t!e pro5ect9release
2010 Holl
Consulting
+,trapolation to )elease -ate
#i(e
C
u
(
u
l
a
t
i
*
e

-
e
f
e
c
t
s
Pro.ect Defects
#otal
#argete.
#o QA
Close.
)elease #arget -ate
S!oul. " ;e
concerne.%
2010 Holl
Consulting
+,trapolation to )elease -ate
#i(e
C
u
(
u
l
a
t
i
*
e

-
e
f
e
c
t
s
Pro.ect Defects
#otal
#argete.
#o QA
Close.
)elease #arget -ate
!en will t!e
pro5ect ;e .one%
2010 Holl
Consulting
Co(puting in +,cel
Query .efects into +,cel
Con*enient8 Vse na(e. ranges for .ata
2010 Holl
Consulting
Co(puting in +,cel
&
"n a secon. s!eet, create a .ate colu(n an. count
t!e .efects su;(itte. an. close. for eac! .ate.
Array for(ula. QS!iftPCtrlP)eturnR
2010 Holl
Consulting
)eal 3ife +,a(ple
"ard*ar9 5(@ Defect Disco*ery & >losure
>it! e,trapolations?
0
B0
100
1B0
200
2B0
E00
EB0
400
4B0
B00
A
9
1
F
9
0
1
A
9
2
4
9
0
1
F
9
1
9
0
1
F
9
1
9
0
1
F
9
1
B
9
0
1
F
9
2
2
9
0
1
F
9
2
9
9
0
1
1
9
B
9
0
1
1
9
1
2
9
0
1
1
9
1
9
9
0
1
1
9
2
A
9
0
1
8
o
t
a
l

>
2
s
0
10
20
E0
40
B0
A0
F0
10
90
100
A
o
u
n
d

i
n

5
(
@
J Tpen
J Close.
J :oun. in 4.F
A91A G Start of testing
#arget
)elease
-ate
2
n.
or.er polyno(ial fit
2010 Holl
Consulting
)elease Criteria8 -efects
&
$oals8
'ro*i.e goo.Penoug! 4uality to custo(ers

&
-on/t release any M;a.N .efects
&
-on/t release Mtoo (anyN .efects
2ini(iLe support costs

&
Custo(ers will nee. support proportional to
pro.uct 4uality
2010 Holl
Consulting
-efect -isco*ery
#i(e
C
u
(
u
l
a
t
i
*
e

-
e
f
e
c
t
s

:
o
u
n
.
)elease #arget -ate
S!oul. " release%
2010 Holl
Consulting
Ha. -efects
&
Ha. is a function of
e*erity G t!e conse4uences of t!e failure
Pro,a,ility t!at it will occur
#!e num,er of customers li6ely to e,perience it
2ay also inclu.e Mi(portantN custo(ers or functions

&
#!ese are often co(ple, to calculate, so (ost
co(panies 5ust use Se*erity, te(pere. ;y ot!er
factors
2010 Holl
Consulting
Quality #!res!ol.
&
Io (ore t!an t!e following nu(;er of open
.efects8
!ere Iew G .efects .isco*ere. in t!e current pro5ect, an.
Tl. G .efects .isco*ere. in pre*ious releases

e*erityB :ew 3ld
Critical 0 0
"(portant B 10
2o.erate E0 100
#ri*ial B0 200
Release team (stakeholders) need to
determine values based on history. The
goals are:
& Not to release any Critical defects
& Not to introduce too many ne
defects
& !anage the backlog
2010 Holl
Consulting
!at/s $oo. +noug!%
The two market leaders Quicken
TurboTax, produced by Intuit, and
TaxCut, produced by H&R Block
It also oers better end!
o!return re"iews #
TurboTax oers better "ideos than
TaxCut #
2010 Holl
Consulting
!at/s $oo. +noug!%
TaxCut oers easier ta$$in$ o
items you%d like to deal with later&
It also appears to be more
scrupulously prooread than
TurboTax' Twice in TurboTax%s
screens in my tests, incorrect
inormation was displayed, thou$h
the tax returns themsel"es were
unaected&
2010 Holl
Consulting
!at/s $oo. +noug!%
# i you%re choosin$ which pro$ram
to $o with or the irst time this year,
I%d $o with TaxCut solely because
o the text errors in TurboTax&
I ha"e a hi$her standard or accuracy in tax
pro$rams than I do or any other type o
sotware, and while the errors we ound didn%t
aect the bottom line, they did aect my
conidence in the pro$ram%s makers&
It%s unortunate, because I elt
TurboTax oered better eatures&
2010 Holl
Consulting
-etroit :ree 'ress
R 1.F2 )ea.ers
X we; site
X wire ser*ice
#!e article ca(e fro( t!e
Monterey Peninsula Herald
2010 Holl
Consulting
're.icting -efects
&
#!ere are se*eral (o.els for pre.icting
.efects
&
#!e pre.ictions of interest are8
How (any .efects are going to ;e create.%
!en will " fin. t!e(%
How long will it ta6e to fi, t!e(%
!en will " ;e .one%
How goo. will t!e result ;e%

2010 Holl
Consulting
Vsing -efect -ensity
&
'!ase -efect -ensity
--
'!ase
G
&
At first t!is (ay not see( rig!t, ;ut it/s a reasona;le
pre.ictor
&
!ic! is (ore relia;le%
AF specification .efects per 200 pages of .ocu(entation

>0.E4 .efects9page?
AF specification .efects per 4E00 S3TC >1B.A .efects9US3TC?
-efects "n5ecte.
'!ase
#otal 'ro5ect Affecte. S3TC
2010 Holl
Consulting
Vsing -efect -ensity
&
'ro5ect9'rocess -efect -ensity
Iu(;er of .efects per

affecte. S3TC
Affecte. S3TC can ;e

.ifficult to get wit!out a
goo. tool.
#!e ;est "/*e foun. is

Co.e)eports
>a6a Source)T"?
!ttp899www.polariguana.co(9sourceroi.p!p
2elease " 2elease <
Defects
4B2 BAE
8otal C3>
1,A00 4E,B00
Defect
Density
B2.A 12.9
Pro.ect
>hanged
C3>
B,400 4,EB0
Pro.ect
Defect
Density
1E.F 129.4
2010 Holl
Consulting
Calculating -- per '!ase
Y -efects "n5ecte. 'er '!ase
'ro5ect US3TC
Iu(;er of -efects In.ection Phase
)e4uire(ents 4A 0 0 0 0
-esign 1 2A 0 0 0
Co.e 1E 19 1E 0 0
#esting EE 22 EF 2 0
'ro.uction E 2 1A 0 1
8otal In.ected 1FE A9 1EA 2 1
Defect Density 5(95 1(9@ 3(D9 0(06 0(03
'ro5ect
S3TC G
EB,000
Iote8 if t!is is
only test .efect
.ata, you/re
(issing an
opportunity to get
;etter
infor(ation.
--
'!ase
G
2010 Holl
Consulting
Ie,t 'ro5ect
&
+sti(ate. to ;e 20,000 S3TC
Eistorical
Defect
Density
Ci9ely
to
In.ect
Phase
>ontainment
Ci9ely
to Cea9
1ffort to
2esol*eB
8otal
2ewor9
2e&uirements 4.94 99 2A.AC 2A 1B.E 2242.2
Design 1.9F E9 1F.EC F E1.B 2AE.E
>ode E.19 F1 A2.1C 41 11.E 112.9
8esting 0.0A 1 11.FC 1 2.2 2.1
Production 0.0E 1 100.0C 1 1 0.A
#otal 211 Frand 8otal 3391(0
D orst case cost >.efects .isco*ere. in test?
2010 Holl
Consulting
-efect -isco*ery S Closure
&
're.ictions can ;e
(a.e in +,cel ;y
e,trapolating an.
seeing w!ere t!e
.isco*ery an.
close lines (eet
>umulati*e Defects u,mitted
0
B0000
100000
1B0000
200000
2B0000
E00000
1
0
9
1
4
9
2
0
0
A
1
0
9
2
1
9
2
0
0
A
1
1
9
1
1
9
2
0
0
A
1
1
9
2
B
9
2
0
0
A
1
2
9
9
9
2
0
0
A
1
2
9
2
E
9
2
0
0
A
1
9
A
9
2
0
0
F
1
9
2
0
9
2
0
0
F
2
9
E
9
2
0
0
F
2
9
1
F
9
2
0
0
F
E
9
E
9
2
0
0
F
E
9
1
F
9
2
0
0
F
E
9
E
1
9
2
0
0
F
4
9
1
4
9
2
0
0
F
4
9
2
1
9
2
0
0
F
B
9
1
2
9
2
0
0
F
B
9
2
A
9
2
0
0
F
A
9
9
9
2
0
0
F
A
9
2
E
9
2
0
0
F
F
9
F
9
2
0
0
F
F
9
2
1
9
2
0
0
F
1
9
4
9
2
0
0
F
2010 Holl
Consulting
+,trapolation
&
#!is (et!o. is
4uic6 an. in
so(e cases
surprisingly
goo.
&
Hut not in all
cases
2010 Holl
Consulting
Tt!er +,cel +,trapolations
&
A..ing a tren. line uses all t!e .ata
&
Zou (ay only want to use some of t!e
.ata
&
Create a parallel colu(n of t!e .ata you
want to e,trapolate an. e,trapolate only
t!at cur*e
2010 Holl
Consulting
+,a(ple
2010 Holl
Consulting
Si(ple 3inear +,trapolation
&
#o create a linear e,trapolation on t!e last
few .ata points, start ;y e,ten.ing t!e
.ate
colu(n8
2010 Holl
Consulting
&
Select t!e cells t!at
loo6 linear
&
Copy an. 'aste
Special P Oalues
2010 Holl
Consulting
&
#!en .rag t!e cells .own to t!e last
e,trapolate. .ate
&
+,cel will
.o a linear
e,trapolation
of t!e .ata
2010 Holl
Consulting
+,trapolation
&
"n or.er for e,trapolations to ;e relia;le,
t!e .isco*ery rate (ust !a*e pea6e.
$ee9ly Defects u,mitted
0
10
20
E0
40
B0
A0
1
9
1
9
2
0
0
1
1
9
1
B
9
2
0
0
1
1
9
2
9
9
2
0
0
1
2
9
1
2
9
2
0
0
1
2
9
2
A
9
2
0
0
1
E
9
1
1
9
2
0
0
1
E
9
2
B
9
2
0
0
1
4
9
1
9
2
0
0
1
4
9
2
2
9
2
0
0
1
B
9
A
9
2
0
0
1
B
9
2
0
9
2
0
0
1
A
9
E
9
2
0
0
1
A
9
1
F
9
2
0
0
1
F
9
1
9
2
0
0
1
F
9
1
B
9
2
0
0
1
F
9
2
9
9
2
0
0
1
1
9
1
2
9
2
0
0
1
1
9
2
A
9
2
0
0
1
9
9
9
9
2
0
0
1
9
9
2
E
9
2
0
0
1
1
0
9
F
9
2
0
0
1
2010 Holl
Consulting
-one%
&
!at .oes it (ean to ;e .one >fro( a
.efect perspecti*e?%
Io M;a.N .efects re(ain
're.icta;le9(anagea;le .efects re(ain

&
#wo categories of .efects re(ain
#!ose you 6now
#!ose you .on/t 6now

2010 Holl
Consulting
3atent -efects
&
3atent .efects >un6nown at release ti(e?
can ;e pre.icte. using .efect .ensity an.
)ayleig! '-:
&
!at nee.s to !appen ;efore t!e
pre.ictions ;eco(e relia;le%
#!e .efect .isco*ery rate (ust flatten

d f(t)
dt
G 0
2010 Holl
Consulting
-isco*ery )ate
>umulati*e Defect u,mitted
0
100
200
E00
400
B00
A00
1
9
1
9
2
0
0
1
1
9
1
B
9
2
0
0
1
1
9
2
9
9
2
0
0
1
2
9
1
2
9
2
0
0
1
2
9
2
A
9
2
0
0
1
E
9
1
1
9
2
0
0
1
E
9
2
B
9
2
0
0
1
4
9
1
9
2
0
0
1
4
9
2
2
9
2
0
0
1
B
9
A
9
2
0
0
1
B
9
2
0
9
2
0
0
1
A
9
E
9
2
0
0
1
A
9
1
F
9
2
0
0
1
F
9
1
9
2
0
0
1
F
9
1
B
9
2
0
0
1
F
9
2
9
9
2
0
0
1
1
9
1
2
9
2
0
0
1
1
9
2
A
9
2
0
0
1
9
9
9
9
2
0
0
1
9
9
2
E
9
2
0
0
1
1
0
9
F
9
2
0
0
1
$ee9ly Defects u,mitted
0
10
20
E0
40
B0
A0
1
9
1
9
2
0
0
1
1
9
1
B
9
2
0
0
1
1
9
2
9
9
2
0
0
1
2
9
1
2
9
2
0
0
1
2
9
2
A
9
2
0
0
1
E
9
1
1
9
2
0
0
1
E
9
2
B
9
2
0
0
1
4
9
1
9
2
0
0
1
4
9
2
2
9
2
0
0
1
B
9
A
9
2
0
0
1
B
9
2
0
9
2
0
0
1
A
9
E
9
2
0
0
1
A
9
1
F
9
2
0
0
1
F
9
1
9
2
0
0
1
F
9
1
B
9
2
0
0
1
F
9
2
9
9
2
0
0
1
1
9
1
2
9
2
0
0
1
1
9
2
A
9
2
0
0
1
9
9
9
9
2
0
0
1
9
9
2
E
9
2
0
0
1
1
0
9
F
9
2
0
0
1
2010 Holl
Consulting
:itting -ata to a 2o.el
&
:itting .ata
to a cur*e
!elps us tell
t!e rate is
.ecreasing
2010 Holl
Consulting
Vsing +,cel
&
As ;efore, create a linear e,trapolation of
t!e last QcriteriaR .ays8
&
M:latteningN criteria e,a(ple8
#!e .isco*er rate (ust less t!an B .efects

per .ay in t!e past two wee6s of testing
2010 Holl
Consulting
Vsing +,cel
&
A.. linear tren. line
&
Select M-isplay e4uation on c!artN
2010 Holl
Consulting
Vsing +,cel
&
#!is gi*es t!e slope of t!e line (,
w!ere

yG(, X ;
"n t!is case,

( G 2.2 .efects9.ay
Iot goo. enoug!

>umulati*e Defect u,mitted
y G 2.1AE, P 1BE9B
0
100
200
E00
400
B00
A00
1
9
1
9
2
0
0
1
1
9
1
B
9
2
0
0
1
1
9
2
9
9
2
0
0
1
2
9
1
2
9
2
0
0
1
2
9
2
A
9
2
0
0
1
E
9
1
1
9
2
0
0
1
E
9
2
B
9
2
0
0
1
4
9
1
9
2
0
0
1
4
9
2
2
9
2
0
0
1
B
9
A
9
2
0
0
1
B
9
2
0
9
2
0
0
1
A
9
E
9
2
0
0
1
A
9
1
F
9
2
0
0
1
F
9
1
9
2
0
0
1
F
9
1
B
9
2
0
0
1
F
9
2
9
9
2
0
0
1
1
9
1
2
9
2
0
0
1
1
9
2
A
9
2
0
0
1
9
9
9
9
2
0
0
1
9
9
2
E
9
2
0
0
1
1
0
9
F
9
2
0
0
1
2010 Holl
Consulting
3atent or Unown -efect )is6
&
'ro;a;ility of occurrence in pro.uction
)is6 G Se*erity D 'ro;a;ility

&
Iee. to !a*e a .ecent (easure9esti(ate
of pro;a;ility
Iee. a usage profile

&
Also, fre4uency .oes not always e4ual
*alue
" (ay not use t!e M)estoreN function *ery

often, ;ut it ;etter wor6
2010 Holl
Consulting
An. :inally7
&
#rac6
latent9release.
.efects
carefully
&
#!ey are your
;est in.icator of
Quality
2010 Holl
Consulting
)elease -efect -ensity
:ro(8
oftware Quality
Measurement#
" Aramewor9 for >ounting
Pro,lems and Defects
C2V9S+"P92P#)P022
!ttp899www.sei.c(u.e.u9pu;9.ocu(ents992.reports9p.f9tr22.92.p.f
2010 Holl
Consulting
Agen.a

&
&
2010 Holl
Consulting
A #rue Story
&
A w!ile ;ac6, " loo6e. in t!e (irror an.
.eci.e. " nee.e. to loose so(e weig!t.
&
!at is t!e first t!ing "/( going to .o%
2010 Holl
Consulting
2y :irst Step
2010 Holl
Consulting
2y :irst Step Tn A Scale
&
"f " .on/t (easure (yself to.ay, " won/t
6now
"f t!e c!anges "/( planning to (a6e are ta6ing

(e t!e rig!t .irection
!ic! c!anges !a*e t!e ;est affect
How far "/*e co(e
!en "/( .one

&
Its important to estalish a aseline
2010 Holl
Consulting
!at Are
#!e $oals%
!Would you tell me"
please" which way I
ou#ht to #o from here?$
!%hat depends a #ood deal on where you want to #et
to"$ said the &at'
!I don$t much care where(($ said )lice'
!%hen it doesn$t matter which way you #o"$ said the &at'
2010 Holl
Consulting
Closure -ata Collection
Su;(it reason
& #est case faile.
& 'ass criteria unclear
& #este. wrong *ersion
& Setup9config wrong
& -ataPrelate. error
& Tt!er
Io
& Correction (a.e
& -uplicate
& C!ange not 5ustifie.
"n5ection '!ase
)oot Cause
-efect%
)esolution
Zes
Classification, 2o.e
Correcte. Co(ponent>s?
8his is where we record what we+*e learned
#otal
)esolution
+ffort
)
e
s
o
l*
e
.
2010 Holl
Consulting
-efect Classification 2o.els
&
"n.ustry gui.elines to stan.ar.iLe
infor(ation collection
&
#!e (ost pre*alent are fro(8
"+++
"H2
Hewlett 'ac6ar.
2010 Holl
Consulting
H' -efect Trigins, #ypes, an. 2o.es
2010 Holl
Consulting
-efect 3ea6age )easons
1. +n*iron(ental -epen.ency
2. #est Iot )un
E. #est Case +rror
4. -ata -epen.ency
B. "nconsistent Sy(pto(s
A. #est Case Success Criteria Vnclear
F. #est +,ecution +rror
1. #est Case )esults Iot Oerifie.
9. Co.e C!ange. After #est 'asse.
10. Io #est Case9Co*erage
11. Configuration 2anage(ent 'ro;le(
12. )e4uire(ents 2issing9C!ange.
2010 Holl
Consulting
Vsing 3ea6age -ata
&
:or eac! .efect foun. in pro.uction, log a lea6age
reason
&
Vse t!e fre4uency >*olu(e? to i(pro*e t!e relate.
process
&
)esist te(ptation to weig!t t!e fre4uency ;y
so(et!ing i(portant
#!is assu(es cause S effect, an. t!ere is none
#!is is .ifferent fro( root cause, w!ere t!e cost to repair

is relate. to t!e type of .efect
2010 Holl
Consulting
Specific 'rocess "(pro*e(ent 2etrics
&
#wo types of process (etrics8
1. 2etrics t!at (onitor a process
>tell you t!e effect of c!anges?
& Hasic S'CPtype c!arts
2. 2etrics t!at tell you w!at to .o
& '!ase Contain(ent
&
)oot Cause Analysis
& #est Co*erage
& -efect 3ea6age )easons
& Static Co.e Analysis P Co(ple,ity
2010 Holl
Consulting
S!oul. all pro5ects ;e (easure.%
&
'ros8
2ore .ata points
+*aluate e*eryt!ing we/re .oing

&
Cons8
'ro5ects (ay use .ifferent processes
Oariation (ay ;e too great

&
Suggestions8
-on/t (easure pro*ects (easure processes
Tnly (easure your ;est pro5ects
#ailoring is o6ay, ;ut separate out t!e .ata

2010 Holl
Consulting
Hol. 'erio.ic 'rocess )e*iews
&
2ont!ly Quality )e*iews focuse. on
'rocess "(pro*e(ent initiati*es
2easures S Analysis9"nterpretation
3essons 3earne.
&
)ules of (etrics eti4uette8
're*iew .ata wit! sta6e!ol.ers G no surprises

2010 Holl
Consulting
'!ase Contain(ent
&
".entifies t!e lifecycle acti*ities t!at lea6
.efects
&
#ells you w!ere
to i(pro*e .efect
.etection efforts
Phase >ontainment
0C
10C
20C
E0C
40C
B0C
A0C
F0C
10C
90C
100C
)e4uire(ents -esign Co.e Huil. #esting
6

D
e
f
e
c
t
s

>
o
n
t
a
i
n
e
d
2010 Holl
Consulting
)oot Cause Analysis
&
'areto t!e types of
errors ;eing create.,
weig!te. ;y cost >effort?
2oot >ause 8otal Eours
0
B00
1000
1B00
2000
2B00
E000
)
e
4
u
i
r
e
(
e
n
t
s

A
(
;
i
g
u
o
u
s
9
*
a
g
u
e

)
e
4
u
i
r
e
(
e
n
t
s

2
i
s
s
i
n
g

3
o
g
i
c

f
l
o
w

2
i
s
s
i
n
g

l
o
g
i
c

)
e
4
u
i
r
e
(
e
n
t
s

C
!
a
n
g
e
.

r
o
n
g

*
a
r
i
a
;
l
e

u
s
e
.

T
t
!
e
r

C
a
s
e

s
t
a
t
e
(
e
n
t

#
y
p
o

S
p
e
c
i
f
i
c
a
t
i
o
n

2
i
s
s
i
n
g
$
a
r
;
a
g
e

c
o
l
l
e
c
t
i
o
n

+
,
c
e
p
t
i
o
n

!
a
n
.
l
i
n
g

'
a
r
a
(
e
t
e
r

(
i
s
(
a
t
c
!

)
e
4
u
i
r
e
(
e
n
t
s

r
o
n
g

V
n
i
n
i
t
i
a
l
i
L
e
.

*
a
r
i
a
;
l
e

H
a
.

*
a
l
i
.
a
t
i
o
n

c
!
e
c
6

2
i
s
s
i
n
g

e
n
.

s
t
a
t
e
(
e
n
t

+
n
*

n
o
t

(
a
t
c
!

t
e
s
t
S
p
e
c
i
f
i
c
a
t
i
o
n
P
r
e
4
u
i
r
e
(
e
n
t

(
i
s
(
a
t
c
!
C
o
.
i
n
g

s
t
a
n
.
a
r
.

n
o
t

f
o
l
l
o
w
e
.

S
p
e
c
i
f
i
c
a
t
i
o
n

A
(
;
i
g
u
o
u
s
+
n
*

c
a
p
a
c
i
t
y
S
p
e
c
i
f
i
c
a
t
i
o
n

r
o
n
g
r
o
n
g

(
e
t
!
o
.

c
a
l
l
e
.

-
e
s
i
g
n

l
o
g
i
c

w
r
o
n
g
-
e
s
i
g
n
P
s
p
e
c

(
i
s
(
a
t
c
!
)
e
4
u
i
r
e
(
e
n
t
s

"
n
c
o
n
s
i
s
t
e
n
t

+
n
*

!
a
r
.
w
a
r
e

f
a
i
l
u
r
e
-
o
c

t
y
p
o
-
o
c

*
a
g
u
e
+
n
*

T
p
e
r
a
t
i
o
n
a
l

+
r
r
o
r
"
n
s
t
a
l
l

(
i
s
s
i
n
g

f
i
l
e
s
)
e
4
u
i
r
e
(
e
n
t
s

T
t
!
e
r

-
o
c

i
n
f
o

(
i
s
s
i
n
g
"
n
s
t
a
l
l

w
r
o
n
g

c
o
n
f
i
g
u
r
a
t
i
o
n
"
n
s
t
a
l
l

i
n
c
o
(
p
l
e
t
e
-
o
c

w
r
o
n
g
"
n
s
t
a
l
l

w
r
o
n
g

*
e
r
s
i
o
n
)oot Cause , )ewor6 Hours
2010 Holl
Consulting
Agen.a

&
2010 Holl
Consulting
-efect 'ro;a;ilities
&
ei;ull .istri;utions are useful engineering
(o.els, w!ere t!e tail approac!es Lero
asy(ptotically
&
ei;ull 'ro;a;ility -ensity :unction >'-:? is8
!ere m .eter(ines t!e s!ape of t!e cur*e, an.
c is t!e scale para(eter
t
c
m
t
m
e
t
c
m
(
f(t) +
t
c
m
c
m(,
e
t
c
m
(
+
>i6ipe.ia?
2010 Holl
Consulting
-ifferent ei;ull -istri;utions
$ei,ull >ur*es
0
1
2
E
4
0 1 2 E
(G0.B
(G1
(G2 (G4
(G10
ei;ull cur*es wit!
.ifferent s!apes
>*alues of (? can
;e use. to (o.el
*arious syste(s
>pro;a;ility
.istri;utions?
2010 Holl
Consulting
ei;ull 'ro;a;ility -istri;ution
$ei,ull >ur*es
( G 0.1 to B.0 in .1 incre(ents
0.0
0.B
1.0
1.B
2.0
0.0 0.B 1.0 1.B 2.0
2ayleigh %m;4)
2010 Holl
Consulting
ei;ull Cur*es
$ei,ull >ur*es
( G 0.1 to B.0 in .1 incre(ents
0.0
0.B
1.0
1.B
2.0
0.0 0.B 1.0 1.B 2.0
2ayleigh %m;4)
%almost) :ormal %m;3(5)
2010 Holl
Consulting
)ayleig! 'ro;a;ility -istri;ution :unction
&
ei;ull .istri;ution wit! s!ape >(? G 2
&
3et - G 2t
t
c
2
t
2
e
t
c
.
(
f(t) +
2t
c
2
e
t
c
2
(
+
-
c
2
e
-
2c
.
(
f(-) +
-
c
.
e
-
2
4c
2
(
+
2010 Holl
Consulting
)ayleig! 'ro;a;ility -istri;ution :unction
-
c
2
e
-
2
4c
2
(
f(-) +
2ayleigh >ur*es
c G 0.1 to 0.B
0
1
2
E
4
B
A
F
1
9
0.0 0.B 1.0 1.B
c G 0.1
c G 0.2
c G 0.E
2010 Holl
Consulting
+,cel :or(ula
2010 Holl
Consulting
Applying )ayleig! '-:
& 3et t
m
G ti(e at w!ic! t!e cur*e reac!es its pea6
&
#o fin. t!e pea6, we set G 0
& Sol*ing gi*es t
m
G
&
#o scale t!e area un.er t!e cur*e, (ultiply ;y K
& Su;stitute c = t
m
2
d f(t)
dt
c
2
2010 Holl
Consulting
:itting #!e )ayleig! '-:
2t
c
2
e
t
c
2
(
f(t) + K
2t
(t
m
2?
2
e
t
>t
m
2?
.
(
+ K

2t
2t
2
m
e
( t
2
2t
2
m
+ K

1
t
m
te
( t
2
2t
2
m
f(t) + K
2
"f we can esti(ate !ow (any .efects we/ll fin. >K? an. w!en
we/*e !it t!e pea6 >t
m
?, we can create a '-: for our pro5ectW
2010 Holl
Consulting
Huil.ing Zour +sti(ate
2010 Holl
Consulting
're.icting 3atent -efects
&
At pea6, t!e area un.er t!e )ayleig! cur*e is E9.EBC
&
#!is (eans A0.ABC of t!e .efects are yet to fin.
&
+,a(ple8
At pea6 you foun. a cu(ulati*e total of B1 .efects.
#otal G >B19E9.B?D100G14F.4
14F P B1 G 19 re(aining.
3atent -efects + K dt
1
t
m
te
( t
2
2t
2
m
2
)elease -ate
1 year%
2010 Holl
Consulting
+asier t!at Calculus
&
"n +,cel, run a colu(n
for cu(ulati*e total an. a
colu(n for re(aining
.efects
&
:ro( your release .ate
you can easily esti(ate
latent .efects
2010 Holl
Consulting
Cu(ulati*e )ayleig! 2o.el
t
c
2
(
C-: G 1 P e
2010 Holl
Consulting
)ayleig! Assu(ptions
&
#!e .efect rate o;ser*e. .uring t!e
.e*elop(ent process is positi*ely
correlate. wit! t!e .efect rate in t!e fiel..
&
$i*en t!e sa(e error in5ection rate, if
(ore .efects are .isco*ere. an. re(o*e.
earlier, fewer will re(ain in later stages.
2010 Holl
Consulting
+,ponential 2o.els
&
+,ponential 2o.el >ei;ull, (G1?
,
c
e
t
c
(
f(t) +
1G!onential
ei;ull, ( G 1
0.0
0.2
0.4
0.A
0.1
1.0
1.2
0.0 0.B 1.0 1.B 2.0
& +rror .etection rate
& "nstantaneous failure rate
& HaLar. rate
& "n general, t!is is a
reliaility #rowth model
2010 Holl
Consulting
:itting A 2o.el to -ata
&
+asy:it fro( www.(at!wa*e.co(
2010 Holl
Consulting
2o.el Su((ary
&
)ayleig! is ei;ull .istri;ution wit! (G2
&
Tt!er *alues of ( pro.uce ot!er
pro;a;ility .istri;utions
&
#!ere are (any (o.els= try se*eral t!at fit
&
/)ll models are wron#' Some models are
useful'0
$eorge +. Ho,
2010 Holl
Consulting
Su((ary
&
A .efect (anage(ent pro#ram
inclu.es
A ;alance. solution
#raining S e,plaining
2etrics t!at are use.

Questions
?
C!ris Holl
C!ris[Holl\!ot(ail.co(
:eGt >lass# Tools for Metrics

Defect Management & Metrics: Society For Software Quality April 24, 2010

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Defect Management & Metrics: Society For Software Quality April 24, 2010

Uploaded by

Copyright:

Available Formats

Defect Management &

Su((ary of 're*ious #al6

<ust log functional pro;le(s foun. in test= .e*iations fro( t!e

Cost of a re4uire(ents .efect G 1

Cost of a .esign .efect G 10

Cost of a co.ing .efect G 100

Cost of a pro.uction .efect G 1000

Su((ary of 're*ious #al6

-efect 2anage(ent 2etrics

#!ere are two answers

Iu(;er of 3!erational .efects .isco*ere.

Iu(;er of Minor .efects foun.

Ha*e clear gui.elines for Tperational an.

Si(ilar to esti(ate .escri;e. wit! re*iews

)e(o*e wee6en.s to s(oot! t!e cur*e

#est co*erage is goo.

Iot continuing to a.. new test case

'ro*i.e goo.Penoug! 4uality to custo(ers

2ini(iLe support costs

e*erity G t!e conse4uences of t!e failure

Pro,a,ility t!at it will occur

#!e num,er of customers li6ely to e,perience it

2ay also inclu.e Mi(portantN custo(ers or functions

!ere Iew G .efects .isco*ere. in t!e current pro5ect, an.

Tl. G .efects .isco*ere. in pre*ious releases

How (any .efects are going to ;e create.%

!en will " fin. t!e(%

How long will it ta6e to fi, t!e(%

!en will " ;e .one%

How goo. will t!e result ;e%

AF specification .efects per 200 pages of .ocu(entation

Iu(;er of .efects per

Affecte. S3TC can ;e

#!e ;est "/*e foun. is

Io M;a.N .efects re(ain

're.icta;le9(anagea;le .efects re(ain

#!ose you 6now

#!ose you .on/t 6now

#!e .efect .isco*ery rate (ust flatten

#!e .isco*er rate (ust less t!an B .efects

"n t!is case,

Iot goo. enoug!

)is6 G Se*erity D 'ro;a;ility

Iee. a usage profile

" (ay not use t!e M)estoreN function *ery

Su((ary of 're*ious #al6

-efect 2anage(ent 2etrics

2etrics for 'ro5ect 2anage(ent

"f t!e c!anges "/( planning to (a6e are ta6ing

!ic! c!anges !a*e t!e ;est affect

How far "/*e co(e

!en "/( .one

#!is assu(es cause S effect, an. t!ere is none

#!is is .ifferent fro( root cause, w!ere t!e cost to repair

2ore .ata points

+*aluate e*eryt!ing we/re .oing

'ro5ects (ay use .ifferent processes

Oariation (ay ;e too great

-on/t (easure pro*ects (easure processes

Tnly (easure your ;est pro5ects

#ailoring is o6ay, ;ut separate out t!e .ata

'rocess "(pro*e(ent initiati*es

're*iew .ata wit! sta6e!ol.ers G no surprises

)e(oe wee6en.s to s(oot! t!e cure

Tl. G .efects .iscoere. in preious releases

+aluate eeryt!ing we/re .oing

'rocess "(proe(ent initiaties