Software Project Management

Software Measurement Overview

 Agenda
• Introduction to Measurement
• When do we Measure?
• Why Measure?
• Terminology: Measure and Metric
• Terminology: Metrics
• Measurement Principles
• What to Measure - Examples
• How to Measure
• How to Analyze What You’ve Measured
• Common Measurement Mistakes
• Measuring Resources
 When Do We Measure?
• Launching, Planning & Executing
• Estimating the project
• Defining metrics collection & reporting mechanisms
• Executing & Controlling
• Tracking progress and taking corrective action as appropriate
• Closing
• Storing historical data
 Why Measure
• To establish realistic project expectations.
• So you can control the project and communicate status
and progress.
• To understand the software process.
• To focus people’s activities on the process.
• To improve morale by bringing attention to chronic
problems.
• To improve business performance.
• To lay the foundation for long-term improvement.
 Terminology: Measure & Metric
• Measure
• The size or extent of something, especially in comparison
with a known standard. For example:
• Number of defects
• Thousand Source Lines of Code (KSLOC)
• Metric
• A calculated or composite indicator based on two or
more measures.
• A quantified measure of the degree to which a system,
component or process possesses given attributes. For
example:
• Number of defects per KSLOC
 Terminology: Metrics
• Enable us to
• Estimate project parameters
• Measure progress on meeting an objective
• Focus on the vital few
• Balance our approach to
• Project Progress
• Product Quality
• Processes
• Customer Satisfaction
• Organizational Readiness
• Financials
 Terminology: Metrics 2
• A combination of two ormore metrics gives information meaning. For
example, compare:
• 10 defects/KSLOC (Is this good? Bad?)

Versus

• Release 1- 14 defects/KSLOC
• Release 2- 10 defects/KSLOC
• Release 3- 8 defects/KSLOC
(The trend is clearly good.)
 Useful Metric Characteristics
• Measurable
• Anything unmeasured, but considered to be important, is called an ‘unquantum.’
• “The principle use for unquantais rationalization.”
• Independent
• Of conscious influence by the project team.
• No way to change the value other than by changing the underlying quality it
reflects
 Useful Metric Characteristics - 2
• Accountable
• Don’t put off analysis ofthe data you collect.
• If gaps are found, it’s too late to re-collect.
• If you draw conclusions from the data, then save the data and analysis process.
• Precise
• Goal isn’t to maximize precision, but rather to note it.
• Helps avoid using the data later on in ways that are inconsistent with the accuracy.
 Measurement Principles
• Make sure you are measuring what is useful.
• You can not
• Improve without a baseline
• Establish a baseline without measurements
• Create measurements without understanding the process
• Control what you can’t measure
• Metrics
• Must be tied to business objectives
• Exist to measure project progress
• Must include goals and timeframe
• Must be clearly and consistently defined
• Must be used by management and customers
• Must trigger action plans when progress doesn’t match expectations
• Must neverbe used to punish individuals
Business Objectives
 What to Measure - Objectives
• Project’s Business Objective: Improve the performance of the XYZ
product so that the customer can reduce their design time by 10%.
• What do we want to achieve?
• Increase product performance by 15% from previous release.
• What do we need to measure?
• XYZ product performance.
• Customer design time when using XYZ.
 What to Measure - Objectives (2)
• Project’s Cycle Time: Ensure that the MNO project completes within 10%
of their completion date estimated at the end of design.
• What do we want to achieve?
• Ensure project completion within 10% of the date estimated at the end of design.
• What do we need to measure?
• Size, schedule & budget estimates
• Size, schedule & budget actuals
• Project scope changes
 Goal – Questions - Metrics
• Measurement defined intop down fashion
 Possible Business Objectives
• Goal: Project’s Business Value
• Question: Did the project’s actual business value (financial and/or strategic) match
the estimated business value? Why or why not?
• Metrics
• Estimated project value
• Estimated investment
• Actual project value
• Actual investment
 Possible Business Objectives
• Goal: Track Project’s Cycle Time Performance
• Question: Do the project’s actual completion dates match the estimated
completion dates? Why or why not?
• Metrics
• Estimated completion dates by phase, task and deliverable.
• Estimated effort by resource by task and deliverable.
• Actual completion dates by phase, task and deliverable.
• Actual effort by resource by task and deliverable.
• Log of key events, changes and/or decisions
• impacting project performance.
 Possible Business Objectives -2
• Goal: Track Product Quality
• Question: Is the product’s quality sufficient to warrant release? Why or why not?
• Metrics
• The total number of test cases available by build.
• The total number of executed test cases by build.
• The number of new defects found by build.
• The number of open defects versus the total defects by build (to help
predict release dates).
• The number of pre-release defects removed from the product (to
assess product quality).
• The number of post-release defects found (to calibrate our estimating
abilities).
 What to Measure – Suggestions
• Caveats
• Each organization needs to determine what to measure based on its own
objectives.
• Tie measures to goals.
• Don’t measure if you aren’t going to use the data.
• Resource Data
• Effort by activity, phase and skill.
• % resource needed by calendar time.
• Cost Data
• By resource.
• By activity.
• Schedule Data
• Duration by activity, phase and skill.
 What to Measure – Suggestions-2
• Change and Defect Data
• Defects by classification
• Pre- or post-release, artifact, severity, status, root cause, phase found,
phase introduced
• Effort to detect & correct each defect
• Product and project changes over time
• Time Accounting Data
• To the nearest ½ hour
• By WBS
• By activity
• By person
• Every week (or appropriate time period)
 What to Measure – Suggestions-3
• Product Data
• Product type
• Shrink wrap, web, business, etc.
• Functions, use cases, and/or classes
• Development dates
• Total effort
• Project resources and roles
• Process Data
• Process definition
• Process effort and/or duration
 Analyzing Your Data
• Validate conformance to plan.
• Confirm a theory.
• Example: Implementing an inspection process will reduce
project cycle time and defects by 10% without raising costs.
• Explore a relationship so you have a baseline to
compare for the future.
• Example: What are the ranges of quality for my projects?
• Perform what-if analysis.
• Given initial process execution data, what will the result be if
we continue to execute without hange?
How to Analyze What You’ve
Measured – Example 1a
How to Analyze What You’ve
Measured – Example 1b
 Common Measurement Mistakes
• Failureto set targets
• “If you don’t care about quality, you can achieve any other
objective.”
- Zeroeth law of software engineering.
• Failureto set measurable targets. Are goals met when
the project completes?
• “User Friendly”, “Reliable”
• Failureto quantify component costs
• Project had cost overrun, but why?
• Requirements? Design? Coding? Test?
• Failureto quantify product quality
• What should the customer expect?
 Common Measurement Mistakes (2)
• Capturing more data than needed/useful:
• We don’t balance usefulness with cost/effort.
• Mistaking opinions for metrics:
• A metric is not merely an opinion expressed in numeric
terms.
• Not acknowledging that collecting and using metrics
takes time, effort and money.
• Not recognizing Heisenberg’s Uncertainty Principle
(slightly revised by DeMarco).
• “Measuring any project parameter and attaching evident
significance to it will affect the usefulness of that parameter.”
 Common Measurement Mistakes (3)
• Failure to question claims “supported by data.”
• How were results obtained?
• Not or partially collecting data.
• E.g., if all project effort isn’t collected then your resulting
metrics will be inaccurate.
• Not discussing the data collection goals, forms and
mechanisms with the team to gain support.
• Not discussing the collected data with the team to gain
insight and support.
 Measuring Resources
• Human Resources
• Estimated vs actual full-time equivalent (FTE) headcount over time.
• Estimated vs actual effort using time-accounting.
• Participant logs effort hours against scheduled and/or unscheduled activities.
• Other Resources (small for software projects)
• Capital equipment
• Hardware, software, infrastructure, …
Sample Resource Metrics
 Summary
• Organizations that have established full software-
measurement programs have often improved quality
by about 40% per year, and productivity by 15% per
year for 4-5 consecutive years – Capers Jones.
• Organizations with accurate measures of software
defect rates and defect removal tend to dominate their
industries – Capers Jones.
• The cost for this level of improvement is typically 4-5%
of the total software budget.
Questions & Answers
Software Measurement Overview