Facilitating your process journey …

A Practical Approach to Defining Useful Measures
Houston Texas Chapter of the IIBA Houston, TX August 7, 2008

Facilitating your process journey …

Agenda

Measurement and Analysis Process Overview Plan Measurement Perform Measurement Evaluate Measurement
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 2

Facilitating your process journey …

Measurement Process Model
Objectives and Issues

Technical and Management Processes

User Feedback Analysis Results

Core Measurement Activities Plan Measurement
Measurement Plan New Issues

Perform Measurement
Analysis Reports and Performance Measures

Improvement Actions

Evaluate Measurement

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

3

Facilitating your process journey …

Role of Measurement
The purpose of measurement is to provide the proper objective information needed to make informed decisions that affect the business and/or project outcome

Aids early problem identification

Supports making informed decisions and justifying the decisions

Supports tracking of progress to achieve specific goals and objectives

Supports evaluating process and project performance

Enables effective communication of process and project performance results

Most effective when implemented in support of an organization’s business and technical objectives and when integrated with existing technical and management activities that define a project
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 4

Facilitating your process journey …

Fundamental Measurement Principles - 1
1. Measurement and Analysis is a process

2. Information needs define the measurement objectives which in turn define the measures that support the information needs

3. Processes define the measures that support the measurement objectives
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 5

Facilitating your process journey …

Fundamental Measurement Principles - 2
4. Processes are appropriately “instrumented” to allow data collection

5. Measures are analyzed to ensure that only necessary data are collected to address the measurement objectives

6. Measures are used to aid the decision making process
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 6

Facilitating your process journey …

Fundamental Measurement Principles - 3
7. Analysis results are interpreted within the context of other information and data (integrated analysis)

8. Measurements provide objective information to facilitate effective communication

9. Measurement and Analysis techniques are applied to the Measurement and Analysis process
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 7

Facilitating your process journey …

Measurement and Analysis Process
Principle #1 Measurement and Analysis is a process
Objectives and Issues

Technical and Management Processes

User Feedback Analysis Results

Core Measurement Activities Plan Measurement
Measurement Plan New Issues

Perform Measurement
Analysis Reports and Performance Measures

Improvement Actions

Evaluate Measurement

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

8

Facilitating your process journey …

Issue-Driven Measurement
Principle #2 Information needs define the measurement objectives which in turn define the measures that support the information needs

Information Needs

Measures

Objectives, Issues

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

9

Facilitating your process journey …

Measurement Framework
Goals and Objectives

Corporate Focus
Senior Management

Performance Measures Customers Process Measures Measurement Repository •Estimates •Quality data •Peer Review data •Customer Satisfaction data •Etc.

Process Focus

A

B

C

Process Data

Project /Product Focus
Project 1 Project 2 Project 3 Project 4

Project / Product Data

Historical Measures
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 10

Facilitating your process journey …

Measurement and the Process
 Process characteristics will drive:  what can be measured  how measures are generated and captured
Principle #3 Processes define the measures that support the measurement objectives

 Measurement should be a by-product of process execution

Coding Activity

Testing Activity

Example Base Measures
8/5/2008

1. lines of code 2. complexity

1. # of defects found 2. defect type
11

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

Facilitating your process journey …

Agenda

Measurement and Analysis Process Overview Plan Measurement Perform Measurement Evaluate Measurement
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 12

Facilitating your process journey …

Plan Measurement
Identify and prioritize objectives/issues Describe information needs and associated measurable concepts Design the measures to address each information need

Develop the measurement procedures to collect and organize the data

Build the measurement plan, integrating measurement into project processes

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

13

Facilitating your process journey …

Identify and Prioritize Objectives/Issues
 Each project and organization has unique information

needs  When identifying issues, consider sources such as
 constraints and assumptions  project or organization characteristics • domain, technology • people and skills • product features and acceptance criteria • cost, schedule, and milestones  risks

 past experience of the team
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 14

Facilitating your process journey …

Describe the Information Need
 For each top rated objective or issue, describe the information need  who needs to know what information in order to make what decision  Results from measurement must  be useable by the person or group (who)  answer the need (know what information)  enable the action (make what decision)  Describe a measurable concept for each

need

 what data might be used  gathered and analyzed at what times  reported in what ways

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

15

Facilitating your process journey …

Example of a Common Information Need
 Issue: Project schedule appears to be very aggressive Information Needs:  The Project Manager and team members need to regularly determine which tasks are late, to be able to make corrections in the plan.  The Project Manager needs to know the cumulative effects of task slippage during design, code, and test, to know when a significant problem occurs.
Collect and analyze data at a level of detail sufficient to identify and isolate problems.

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

16

Facilitating your process journey …

Objectives and Information Needs
Objective Requirements need to be managed better
since volatile requirements are causing projects to fail.

Information Need
Example: The Project Manager needs to know the number of requirements changes that the project can handle without additional schedule or budget increases.

Measurable Concept
Example: At the end of the week, the Requirements Analyst reports to the Project Manager the current requirements stability, total and by type of change. The Project Manager investigates the requirements growth for the week and the month and modifies the plan as appropriate.

AS TEAMS, DEVELOP AN EXAMPLE INFORMATION NEED AND CONCEPT USING YOUR HANDOUT

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

17

Facilitating your process journey …

Levels of a Measurement Construct
 Measurement construct captures a chosen approach  identifies levels of measures that address the information need

Attribute
Base Measure Derived Measure Indicator
one or more collections of Measures are used to build an Information Product

Information Product
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 18

Facilitating your process journey …

Leverage Existing Systems
Effort, Schedule, Plans, & Actuals History, Skills Effort/ Schedule Estimates Status Project Management System Milestones, Tasks, Assignments Time Spent Measurement Repository Estimation Tool Effort Actuals

Sizing Estimates, Actuals, Characteristics Counts, Requirements Effort Management System Defects, Rework Defect Reporting System

Project ABC

Time Reporting System

Module and Change Size Configuration Management System Cost, Budget, & Actuals

Status

Budget, Expenditures Financial System

Labor Costs

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

19

Facilitating your process journey …

What Can You Leverage?
Tools
Estimation Models/Methods Requirements Management Time Reporting Project Management Cost Accounting Defect Tracking Code Counting and/or Code Analysis Configuration Management

Available ?

Information Needs It Now Serves

Additional Needs it Might Serve

Test Automation Others?

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

20

Facilitating your process journey …

Integrate Measurement with the Process

Build into process documentation

Build necessary collection and analysis tools

Identify process owner for collection and analysis

Provide training

Use already established meetings and briefings to present and discuss results

Use Quality Assurance function to verify process is followed

Institutionalize collection, analysis, and use

Principle #4 Processes are appropriately “instrumented” to allow data collection
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 21

Facilitating your process journey …

Agenda

Measurement and Analysis Process Overview Plan Measurement Perform Measurement Evaluate Measurement
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 22

Facilitating your process journey …

Perform Measurement
Collect and process base and derived measures

Make decisions and take action

Generate indicators

Analyze Data

These steps occur continuously throughout a project.
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 23

Facilitating your process journey …

Key Concepts
Data need to be captured as close to their creation point as possible

Raw data need to be verified before analysis

Indicators are the building blocks of analysis Analysis is an investigative activity - indicator generation should be flexible enough to satisfy known information needs and to dynamically uncover root causes of problems All analysis involves comparing plan/baseline/expected with actual results
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 24

Facilitating your process journey …

Indicators
 At the reporting periods planned, generate and report

the indicators
 produce standard indicators on regular basis, as planned;

# of Units Completing Unit Test

special indicators as needed  analyze indicators to localize problems  identify new indicators that respond to new questions  collect sets of related indicators

Unit Test Status
140 120 100 80 60 40 20 0

Plan Actual

Project Status Date
Project: OLDMS 8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC Date Prepared: 1 Oct 2009 25

Facilitating your process journey …

Analyze Data
 Analyze the indicators systematically, to assess the

status of known issues and to identify new issues  Have the analysis done by someone independent of those who produce and use the data  While doing the analysis
 identify problems
 assess problem impact  forecast outcome
Principle #5 Measures are analyzed to ensure that only necessary data are collected to address the measurement objectives
26

 evaluate alternatives
8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

Facilitating your process journey …

Integrated Analysis Example for the Information Need: Are we ready to deliver?
Requirements Stability
Number of Requirements
800 600 400 200 0 Plan Actual

Problem Report Status
Number of Problem Reports
700 600 500 400 300 200 100 0

Discovered Closed

Project Status Date
Project: OLDMS

Project Status Date

Design Progress

Date Prepared: 1 Oct 2009

Project: OLDMS

Date Prepared: 1 Oct 2009

# of Units Completed

140 120 100 80 60 40 20 0

Plan Actual

Principle #6 Measures are used to aid the decision making process

Project Status Date
Project: OLDMS

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

Date Prepared: 1 Oct 2009

27

Facilitating your process journey …

Reporting Results
 Use a systematic analysis process as the basis for

reporting results:
   

identify problems assess problem impact forecast outcome evaluate alternatives

Principle #7 Analysis results are interpreted within the context of other information and data (integrated analysis)

 Example:

Problem
• Test planning not started • Development is slightly behind schedule with Component C • Staffing is below planned levels
8/5/2008

Impact
• Test Plan - Medium • Development - Low • Staffing - High

Outcome
• Testing will be late or quality will suffer if planning doesn’t start in the next 2 weeks • If staffing lag continues or worsens, project will be late

Alternatives
• Shift Test Planning to Analysts • Test Component C last • Extend schedule • Limit staff’s nonproject-related commitments
28

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

Facilitating your process journey …

Make Decisions and Take Action
Principle #8 Measurements provide objective understood by the decision makers information to facilitate effective communication  Make results available openly, to all those affected
 The measurement results must be clearly  Action must be taken to realize any benefit from measurement  Desired actions may not be possible - may have to optimize within existing

constraints Principle #9 Measurement and Analysis techniques are applied to the Measurement and Analysis process

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

29

Facilitating your process journey …

Agenda

Measurement and Analysis Process Overview Plan Measurement Perform Measurement Evaluate Measurement
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 30

Facilitating your process journey …

Evaluating Measurement
Evaluate the measures Evaluate the measurement process

Update the experience base

Identify and implement improvements

Like the products and processes it serves, the measurement process periodically needs review and analysis, too
8/5/2008 ®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC 31

Facilitating your process journey …

Summary
Planning Measurement
• identify key issues and information needs • specify measures that assist decision-making

Performing Measurement
Evaluating Measurement
8/5/2008

• collect measures and verify the data • analyze the data and generate indicators • use the measurement results for decision-making

• review the products and process of measurement, to identify improvements

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

32

Facilitating your process journey …

Questions

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

33

Facilitating your process journey …

PPQC Services
Consulting
• • • • CMMI Software Engineering Systems Engineering Process Improvement

Appraising
• SCAMPI A, B, C • Gap Analysis

Training
• • • •
8/5/2008

CMMI/Process Improvement Action Planning Workshops Measurement and Analysis Process Area Specific Training
34

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

Facilitating your process journey …

Contact Information
Address: Henry Schneider President/Senior Principal Consultant Process and Product Quality Consulting, LLC 2111 Heather Green Houston, TX USA 77062
281-218-6682 henry@ppqc.net www.ppqc.net

Phone: E-mail: Web Site:

8/5/2008

®2008 ***Proprietary*** All Rights Reserved - Process and Product Quality Consulting, LLC

35