BIO PRESENTATION

PRESENTATION

T12
Thursday, February 14, 2002
11:30AM

ORTHOGONAL DEFECT
CLASSIFICATION AT CISCO
Bob Mullen
Cisco Systems

International Conference On
Software Management & Applications of Software Measurement
February 11-15, 2002
Anaheim, CA USA
Bob Mullen
The Cisco IOS® operating system is the second most leveraged piece of software in the world.
IOS runs on nearly all of Cisco’s routers, switches, and signal aggregation devices that power
much of the Internet. Bob Mullen is part of the Software Success Engineering team that
continually benchmarks, scopes, (re)defines, implements, measures, and celebrates quality
initiatives to improve Customer satisfaction with Cisco IOS® and related Cisco application
software. He leads the effort to employ orthogonal defect classification to facilitate rapid root
cause analysis. His work provides Cisco the ability to pinpoint the life cycle phase where
defects are injected, guide our defect phase containment efforts, and validate the effectiveness
of our quality improvement initiatives.

Before joining Cisco, Bob served as Director of System Test at Stratus Computer, and before
that was the Manager of Compilers and Commands at Honeywell. Bob has an A.B. from
Princeton and an M.S. from Northwestern. He has also presented his ODC work at ISSRE.
 Orthogonal Defect
Classification at Cisco
Applications of Software Measurement
February 11-15, 2002
Anaheim, CA, USA

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 1
 Orthogonal Defect Classification
(ODC) Deployment
1997 1996

• 4 Cisco software quality engineers attend ODC tutorial

at conference.
• Integrate ODC with one business units’ inspection
process.
• Pilot optional ODC fields in corporate defect tracking
1999 1998

tool for “lead adopter” business unit.

• ISO feedback: need preventive actions.
• Attend ISSRE presentations by IBM, LU, Motorola.
• Write tools for ODC reports via the web.
• ODC fields added to all major software releases trains,
2000

first as voluntary, then required.

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 2
 Tailoring ODC for Cisco

• Minimize additional workload.

• Obtain information from a knowledgeable source.
• Focus on:
Customer impact
What provoked the defect
Life cycle stage where the defect was introduced
Type of error
• Leverage current corporate defect tracking system,
Distributed Defect Tracking Systems (DDTS).
• Use info to drive preventive action at the root cause.

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 3
 Gathering Orthogonal Defect
Classification Information
Keeping It Simple and Speedy

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 4
 Customer Impact

• Focuses on how the defect impacts the customer.

• Categories track to our corporate Customer
Satisfaction Survey.
• Target hardware platforms, software releases, and
technology features that are adversely impacting
customers.
• Entered by the defect submitter.
Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 5
 Defect Trigger

• Identifies the activity that discovered the defect.

• Realigns Development and System Testing with what
customers are doing when they encounter defects.
• Entered by the defect submitter.
Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 6
 Defect Origin

• Identifies where the defect was originally injected.

• Targets the life cycle stage needing process and /
practice improvement.
• Entered by fixer.
Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 7
 Defect Category & Reason

DEFECT CATEGORY DEFECT REASON

• Modifies Defect
Category.
• Identifies the type of
error causing the defect. • Provides additional
focus to the root cause
• Directs the analyst to a of the problem.
corresponding tool,
training, resource, • For example:
process, checklist. Missing Function.
• Entered by fixer. Incorrect Standards.
Extra Message.
• Entered by fixer.

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 8
 Using Orthogonal Defect
Classification Information
Taking Preventive Action

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 9
 ODC Data Transformation
Multiple Report Formats

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 10
 Origin & Category:
Zeroing in on Root Causes of Defects

600 Standards
Feature/Function
Algorithm
500
Data Handling
Error Handling
400
# of Defects

Timing/Serialization
HW/Signal integrity
300
Internal Interface

200 User Interface

Through orthogonal User Doc.
defect classification, 100 U
Us ser D
we’re able to pinpoint Inte er Int oc.
where we are injecting 0 HW rnal I erface
Tim /Sig nterf
n a
defects into the product Err ing/Se al inte ce
or H
ts

rial grit
Co n
en

ig

Da
ta H andlin ization y
Ba Des
m

de
ire

Alg g
Ba e

and
qu

orit
Co

d C ix

ling
Re

se

dF

Fe h m
w

atu
od
Ne

re
tio

Sta /Fu
d
a

nda nct
rte

ce
r
eg

ion
Po

rds
e
ou
Int

ag
uts

are
ck
Pa
O

rdw
ild

Ha
Bu

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 11
 Customer Impact:
Targeting Software Components

Through orthogonal
defect classification,
we can highlight which
software components
are having adverse
impact on customers.

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 12
 Customer Impact:
Internal versus External
Internal
External

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 13
 Orthogonal Defect Classification
Getting to the Root Cause

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 14
 Customer Impact & Triggers:
Realigning our Test Methods

Customers are better at finding

Migration, Maintenance, and
Performance defects than Cisco is.

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 15
 Trigger:
Comparing Component Testing

We can see how effective

testing processes are
by software component.

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 16
 Origin:
Defect Discoveries Over Time

We can see how effective

defect identification
activities are over time.

Cisco Systems
Orthogonal Defect Classification Enhanced Network Architecture Project ASM 2002 – Slide 17
 Cisco Experience

60 – 80+
80
Customer Delivered
Cisco Engineering Hours Expended

Defects are 40 Times More

Expensive Than Removing
60 Defects Early

32 - 50

Ho f
40

s
in t o
ur
al os
ov C
m ng
Re eri
16 - 24

ct e
fe gin
20

De En
2-5
1-3
.2 - .6 .7- 1.2

Requirements Design Code Unit Dev System Field

Testing Testing Testing Maintenance

Life Cycle Phase Where Bug Found

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 18
 Phase Containment of Defects Will
Increase Productivity

At least 3,309 of 11,000 engineers or 30+%

are fixing bugs
3,309 people x $250,000
Annual burdened labor cost is $827,234,250
Engineering best practices can return $345M
of this to Cisco’s bottom line
Conservative cost estimate of the past
year’s bugs:
661,787 hrs = 3,309 person years
of bug fixing effort

Insp/Unit
Insp/Unit Dev
Dev Sys
Sys Cust
Cust
Dev
Dev ARF
ARF Beta
Beta Total
Total
Test
Test Test
Test Test
Test Use
Use

Bugs
Bugs Closed
Closed in
in 1457
1457 56190
56190 93547
93547 48905
48905 4145
4145 6974
6974 25074
25074 236292
236292
the Last Year
the Last Year
Cost
Cost in
in
Engineering
Engineering 22 66 20
20 40
40 40
40 40
40 80
80
Hours/Close
Hours/Close Bug
Bug
Total
Total 2914
2914 337148
337148 1870940
1870940 1956200
1956200 16580
16580 278960
278960 2005920
2005920 6617874
6617874
Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 19
 Static Analysis Tools:
Knowing Where They Are Effective

Through orthogonal
defect classification,
we can see the kinds
of defects that our
static analysis tools
are most effective
at identifying . . .

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 20
 Static Analysis Tools:
Knowing Where They Are Effective

. . . And the kind

customer impact
that their judicious
use can have as a
pay off!

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 21
 Market-Driven Quality Improvement
Closes the Loop on Quality
Customer Listening by Theater,
Network Topology, Technology
Area, Technology Adoption, …

Initiative and goal results

roll up to collective Software Quality Goals
based on collective
improvement demanded customer pain
by customers, coming full
circle

Quality Initiatives at Business Quality Targets at

Unit level – for local Business Unit level
sponsorship and commensurate with
implementation, but leveraging market demands
corporate best practices

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 22
 Contact Information

Bob Mullen
Software Success Engineering
Cisco Systems
300 Apollo Drive
Chelmsford, MA 01824
USA
BoMullen@Cisco.com
IOSQuality@Cisco.com
978-497-8800

Cisco Systems
Orthogonal Defect Classification ASM 2002 – Slide 24