You are on page 1of 47

Metrics and Performance Measurement

System for the Lean Enterprise

Professor Deborah Nightingale

October 24,2005

Overview

• Metrics and Performance measurement

• Why measure? What is performance measure? What are good


metrics?

• Performance measurement and Lean Transformation

• Current practices and Performance measurement


frameworks

• Performance measurement system for the lean


enterprise

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 2
Why Measure?

• “Performance control systems can


serve two purposes, to measure
Strategy
and to motivate.”
- H. Mintzberg,
The Structure of Organizations, 1979
Feed Decision
Back Making
• “The firm becomes what it
measures”
- Hauser and Katz,
You are What You Measure, 2002

Measures Outcomes Actions


• Metrics serve multiple purposes!

Robert Nixon (1990)

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 3
What is performance
measurement?
• Performance measurement is the process of measuring
efficiency, effectiveness and capability, of an action or a
process or a system, against given norm or target.
• Effectiveness is a measure of doing the right job - the extent to
which stakeholder requirements are met.
• Efficiency is a measure of doing the job right - how
economically the resources are utilized when providing a given
level of stakeholder satisfaction.
• Capability is a measure of ability required to do both the job
right and right job, in the short term as well as the long term.
This can be tangible, such as, resources, technology, or
intangible, such as a corporate culture.

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 4
Characteristics of good metrics

• Metrics are meaningful, quantified measures

• Metric must present data or information that allows


us to take action
• Helps to identify what should be done
• Helps to identify who should do it

• Metrics should be tied to strategy and to “core”


processes - indicate how well organizational
objectives and goals are being met
• Metrics should foster process understanding and
motivate individual, group, or team action and
continual improvement.

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 5
A “Good” Metric Satisfies Three
Broad Criteria
1. Strategic
• Enable strategic planning and then drive deployment of the actions
required to achieve strategic objectives
• Ensure alignment of behavior and initiatives with strategic objectives
• Focus the organization on its priorities

2. Quantitative
• Provide a clear understanding of progress toward strategic objectives
• Provide current status, rate of improvement, and probability of
achievement
• Identify performance gaps and improvement opportunities

3. Qualitative
• Be perceived as valuable by your organization and the people involved
with the metric

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 6
Metric Elements

Metric Elements
Explanation
Title
Use exact names to avoid ambiguity

Objective/purpose The relation of the metric with the organizational objectives must be clear
Scope States the areas of business or parts of the organization that are included
Target Benchmarks must be determined in order to monitor progress
Formula The exact calculation of the metric must be known
Units of measure
What is/are the unit(s) used
Frequency The frequency of recording and reporting of the metric
Data source The exact data sources involved in calculating a metric value

The responsible person for performance of that part of the organization,


Owner
collecting data and reporting the metric
Comments Outstanding issues regarding the metric

Adapted from Neely, A., et al. (1995a) Performance measurement system design. 15, 80.

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 7
Role of Performance Measurement

• Monitoring

• Measuring and recording actual performance

• Control
• Identifying and attempt to close the gap between planned target
and actual performance

• Improvement
• Identify critical improvement opportunities

• Coordination
• Information for decision making – Leading Indicators
• Internal communication across processes
• External communication with stakeholders

• Motivation
• Align Behavior and encourage transformation

Source: Vikram Mahidhar

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 8
Transformation to the Lean Enterprise

Functional Enterprise Process Enterprise Lean Enterprise

Value Delivery

Stakeholder Objectives
CEO C Shareholders Customer Community
Executive board U
3 7 3 S Suppliers Employees
6 Processes T
O
5 M
Lifecycle Processes
E
2 4 2
Engineering

Manufact
ring

Pu

uring

R&D
3
Sales
R & s ing

Centers of
rc
e
l es

Manufacturing
i ne

ha

2 Excellence
D
Sa

Enabling Infrastructure
Eng

1 1 1 Processes

Enterprise Leadership &


Governance Processes

Source: Vikram Mahidhar

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 9
Metrics Challenge Today
• Hierarchical organizational architectures giving
way to networked enterprise architectures
• The evolving structure and dynamics of
networked enterprises display immense
complexity
• Metrics response to such complexity has been a
disappointment:
• Hierarchical metrics mindset still continues
• Response to greater complexity has been a metrics explosion

• Challenge: How best to design metrics systems


for networked enterprises?
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 10
Performance measurement system for
Lean Transformation
Knowledge and Behavior Metrics and Assessments
Enterprise impact and results

Transformation over Time


New enterprise capability Industry Government
ORPIC
ROIC
Integration of processes/methods &
tools supporting transformation across
the value stream enabling new
enterprise capabilities Local results and visible indicators
New local behavior Industry Government
Shift in thinking and behavior Cycle time, quality, WIP, on-time
delivery, customer satisfaction,
New routines and ways of doing employee turnover and attitude,
business organizational climate and LESAT
maturity
Organization and group culture change
New approaches (training and Local efforts and new capabilities
introduction of new methods) Industry Government
Engagement in “LAI-venue” with like- Skills, training hours, certification,
minded people lean deployment, joint assessments
and efforts
Enterprise simulation, Lean Now and
LAI knowledge area teams
ROIC = Return on Invested Capital
ORPIC* = Operational Readiness per Invested Capital
Adapted from : Noel Nightingale, 2004
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 11
Generic Enterprise Management Process

Strategy Formulation Strategy Execution


Management Assessment Cascaded Objectives
External - Competitive Intelligence VSM and Project Prioritization
Performance Mgmt Process
Environment - Internal Assessment
Strategic and Operational Planning Communication
Resource Allocation Plan

Value delivery Operations


Shareholder Value
Balanced Score Cards
Customer Satisfaction
Operations Management
Employee Satisfaction
Financial Management
Other stakeholders
Human Resource
Information Systems

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 12
Strategic Metrics

• ROIC (Return on
Invested Capital)
Strategy Formulation
• Economic Value Add Management Assessment
(EVA) - Competitive Intelligence
Strategy Execution
- Internal Assessment

• Net Operating Profit Strategic and Operational Planning


Resource Allocation Plan

• Inventory Turnover
Value delivery Operations
• Revenue
• Cash flow
• Market Position
• Wall Street Expectations

Efficiency metrics and Lagging indicators

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 13
Tactical Metrics

• Financial Turnover

• Budget/Cost and Strategy Execution


Expenses Cascaded Objectives
VSM and Project Prioritization
Strategy Formulation Performance Mgmt Process
• Cost of quality
Communication

• Productivity
Value Delivery Operations

• Supply Chain

Excellence

• Regulatory and social


compliance
Accuracy and timeliness of reporting and control
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 14
Operational Metrics

• Safety
• Quality
• Environment Strategy Formulation Strategy Execution

• Cost/Manufacturing
Efficiency
• Delivery Value Delivery
Operations
Balanced Score Cards
• Time to market Operations Management
Financial Management
• Education and Human Resource

development Information Systems

• Time to Hire

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 15
Value Delivery Metrics

• Stock Price
• Revenue Strategy Formulation Strategy Execution

• On time delivery
• Customer satisfaction
and loyalty Value Delivery Operations

Shareholder Value

• Employee Satisfaction Customer Satisfaction


Competition
Macro Economic Trends
• New product
Introduction

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 16
Time lags - Performance
Management Process

Strategy Formulation Strategy Execution


Management Assessment Cascaded Objectives
External - Competitive Intelligence VSM and Project Prioritization
Performance Mgmt Process
Environment - Internal Assessment
Strategic and Operational Planning Communication
Resource Allocation Plan

Value delivery Operations


Shareholder Value
Balanced Score Cards
Customer Satisfaction
Operations Management
Employee Satisfaction
Financial Management
Other stakeholders
Human Resource
Information Systems

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 17
Time lags - Performance
Management Process

Strategy Strategy Operations Reporting


Formulation Execution

Yearly

Quarter

Month

Day

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 18
Performance Measurement &
Management Frameworks

• Balanced Scorecard – More than 50%


companies have implemented in the US
• Performance Prism

• European Foundation Quality


Framework
• X-Matrix (EVSMA)

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 19
Balanced Scorecard/Strategy Map

Long-Term Growth Strategy


Productivity Strategy
Shareholder Value

Improve Cost Increased Asset Expand Revenue Enhance


Financial Opportunities Shareholder value
Structure Utilization

Customer Price Quality Availability Selection Functionality Services Partnership

Operations Customer Relationship Innovation Process Regulatory and Social


-Supply -Selection -Opportunity Ident. - Environment
Internal -Production -Acquisition - R&D Portfolio - Safety and Health
-Quality -Retention - Design/Development - Employee development
Processes -Logistics -Growth - Time to market - Community

Human capital
Learning
Information capital
& Growth
Culture Organization capital Leadership

Adapted from : Kaplan and Norton, (2000) “Having Trouble with your strategy? Then Map it!” Harvard Business Review Sept_Oct 2000
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 20
Primary Purposes of the
Balanced Scorecard

• Align a balanced set of performance metrics


with business strategy and vision
• Provide management and work teams with
the information necessary and sufficient to
meet their objectives and goals
• Create “line-of-sight” at lower levels of the
organization
• Foster and support process continuous
improvement initiatives

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 21
Desired Characteristics of Performance
Measurement Systems

• Performance Measures should support the

strategic intentions of the organizations

• Managers at all levels should understand


both drivers and results of their activities.

• Explicating Cause-Effect relationships


between drivers and results

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 22
Lean Enterprise Metric
Interdependencies
Process Financial
Quality Yield/Cost Delivery
Health
Customer
Value

Stake holder Value


Processes
Production Logistics
Life Cycle
Infrastructure

Quality
Training
Processes

Control
Enabling

Strategic Change Financial


Leadership

Planning Management Management


Processes
Enterprise

Source: Vikram Mahidhar

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 23
Process Control View –
Performance Measurement System
Strategy
Strategic
Strategic Objectives Decision
Making
Evaluating Plan

Stakeholders Tactical
Targets
Control
Comparison
Function

Operational
Feed back

Action
Output Input
Process
Behavior
Performance
measures

Source: Vikram Mahidhar

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 24
Understanding Performance Measurement
Structures – Causal Models

Illustrative
Example

Metric Cluster

Metric Set

Individual Metric

Source: Vikram Mahidhar

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 25
No One “Right” Set of Metrics

• The balanced scorecard has to be tailored to


each specific company

• The resulting scorecard of indicators should


be driven by the firm’s strategy if it is not to
consist merely of a listing of indicators:

“…although
“…althoughthere
theremay
maybebeaapotentially
potentiallylong
longlist
listof
ofnon-financial
non-financial
indicators,
indicators,individual
individualfirms
firmshave
havetotobe
beselective
selectivebybylinking
linking
explicitly
explicitlytheir
theirchoice
choiceof
ofindicators
indicatorsto
totheir
theircorporate
corporatestrategy.”
strategy.”
R.S. Kaplan and D.P. Norton, Harvard Business Review, January-February, 75-85 (1996)

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 26
Assessing a Performance
Measurement System
• Does it clearly define what constitutes business excellence?
• Does it provide the information required to set aggressive yet
achievable strategic objectives and stretch goals?
• Does it accurately portray our progress and probability of
achieving both long-term strategic objectives and near-term
milestones?
• Does it identify the root causes of barriers?
• Does it focus the organization on the priority improvement needs?
• Does it drive the behavior and actions required to achieve the
objectives?
• Does it align work with value?
• Is it easy to use?
• Does it involve everyone?
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 27
Metrics Will Change Over an Item’s
Life Cycle
Entity Phases Attributes
Business Emerging • Cash flow
Growth • Competitive advantage
Mature • Market share
Declining • Critical Mass
Product Concept • Creative backlog
• Potential product revenue
• Cost per feature
Development • Time to market
• Performance requirements
• Predicted product quality
• Design tocost
In market • Profitability
• Market expansion rate
Phase-out • Volume impact on cost
• Inventory
• Customersupport
Core Recognition • Inventory of skills and capabilities
• Competitive advantage
Competency
Learn • Acquire knowledge
Raytheon Systems, 1998 • Cycles of learning
Practice • Use
• Apply
• Levels of use in organization
• Deployment
Expert • Teach
• Leverage advantage
• Combine and evaluate
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 28
Process and Metric Maturity Model

Process Level Metric


Maturity Maturity

Process management has provided


world-class competitive
advantage (e.g., nodal influence, Holistic
5 Optimizing Metric-driven actions simulated
during strategy setting process to
ensure organizational alignment
agile & forward looking) before metrics are implemented

4
All metrics (process, results,
Support processes are integrated with organizational, geographic, etc.)
and enable core business processes Enabling Total align with strategic objectives,
to provide competitive advantage. Processes Alignment provide competitive advantage
Customer-focused process Integrated & optimize the whole
management is applied
unconsciously Metrics reinforce & leverage

3
Core activities across all core
Common process language & specs Horizontal business processes
Integrated core processes allow a Processes Alignment Local interests are subordinated
seamless flow of work across process Integrated to the good of the whole
boundaries

2
Process metrics added &
Business process management, which Core integrated with result metrics
begins & ends with the customer is
Vertical Metrics aligned between
Processes Alignment strategy & daily activities in
established, in control, and in the Managed
conscious thinking of management. core processes

1
Little or no process focus. That Metrics are ad hoc and
which exists is primarily directed Initial Initial primarily results oriented.
internally toward local operations
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 29
Raytheon Systems, 1998
Level One: Initial

• Enterprise does not manage its business


with a process focus
• Many metrics sub-optimized by local organizational
interests rather than having them aligned with
customer interests and with the strategic
objectives of the enterprise
• Organizations measure the results of past actions

• Results-oriented metrics cannot provide the


leading indicators needed for timely corrective
action to change outcomes

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 30
Level Two: Vertical Alignment
Definition

Vertical
Verticalalignment
alignmentisisthe
theaalignment
lignment and
andre inforcement of
reinforcement ofstrateg ic
strategic
objectives
objectiveswith
withsupportive
supportivegoals
goalsand
andprogress
progressmeasures
measuresat atall
all
levels
levelsofofthe
theorganization.
organization.

• The business enterprise applies a process focus so it can


measure leading indicators of the expected process output
• Defective process output is viewed as a process-capability
problem, not a people problem
• Carefully chosen metrics ensure that all levels of the

organization align with strategic objectives

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 31
Level Two Example

AAcore
core process
process related
related to
to product
product development
development activities
activities
might
mightbe
be documented,
documented,be be in
incontrol
control(repeatable ), be
(repeatable), be
consistently
consistentlydeployed
deployedacross
acrossthe
the organization,
organization,and
andhave
have
measurab le improvement
measurable improvementgains.
gains. IfIfso,
so,that
thatprocess
processis
is
probably
probablyat ator
ornear
nearLevel
Level22 maturity.
maturity. IfIfthe
the metrics
metrics
indicate
indicate variations
variationsin
inthe
the process
processresults,
results,thenthenthey
theyare
are
still
stillat
atLevel
Level11because
because the
the process
processis
isnot
notin
incontrol.
control.

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 32
Level Three: Horizontal Alignment

Two phases:

1. The global optimization of work flow across all process


boundaries. These boundaries become transparent to the flow
of work. Metrics are customer-focused and assess the
enterprise-level capability of a process to provide value from the
customer’s perspective.

2. The global optimization of work flow across all organizational


boundaries that support or use a particular process. Metrics are
customer-focused and assess how well the infrastructure
enables execution of customer-focused processes.

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 33
Level Three Example
The
Theenterprise
enterprisemay
mayhave
haveseveral
severalcore
corecustomer-re lated processes
customer-related processes
such
suchasaswinning
winningnew
newbusiness
businessand
anddeveloping
developingnew
newproducts.
products.AAlso,
lso,
the
theenterprise
enterprisemay
mayhave
havemany
manyfunctions
functionsthat
thatsupport
supportor
orexecute
execute
these
thesecore
coreprocesses.
processes.

Level 3 characteristics include:


• Integrated core processes that customers see as seamless
• Minimized hand-offs or delays as work moves among

processes and sub-processes

• Management focus primarily on early process activities in a


product life cycle
• Metrics insure local organizational interests (functional or
business unit) are subordinated to customer needs and what is
best for the entire business

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 34
Level Four: Total Alignment
Definition

Total
Totalalignment
alignmentis isthe
thesynergistic
synergisticinteraction
interactionofofmetrics
metricsfrom
fromall
all
support
supportprocesses
processeswithwithmetrics
metricsfrom
fromall
allcore
coreprocess
processtotoreinforce
reinforce
the
thestrategy
strategyand
andto
todrive
drivebusiness
businessexcellence.
excellence.

• All employees clearly see where the business is headed


and how they can make a difference.
• Horizontal integration (Level 3) provides employees with
“line of sight” to customer value. Dramatic performance
improvements can occur at this level.
• Total enterprise-level alignment (Level 4) is required to
overcome the major systemic barriers to great performance
and to embed the long-term gains into the fabric of the
organization’s culture.

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 35
Level Four Example

At
At Leve
Levell 4,
4, the
the enterprise
enterprise begins
begins asking
asking how
how
enabling
enabling processes
processes create
create competitive
competitive
advantage
advantage for for the
the core
core customer-re lated
customer-related
processes,
processes, rather
rather than
than what
what they
they do
do to
to improve
improve
themselves.
themselves.

• Total enterprise alignment is required to overcome the major


systemic barriers to great performance

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 36
Level Five: Optimizing

• From a process perspective, the enterprise will have much


greater influence on the market than its size might indicate. The
agile and forward-looking enterprise will be able to foresee
events and respond to those events before they occur.

• From a metrics perspective, the enterprise will be able not only


to simulate and predict the outcome of a strategy before its
deployment, but also to predict the effect of specific metrics on
the outcome of that strategy before choosing metrics.

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 37
Case Study: Nike
• The study focuses on European Operations of Nike.
• The company is organized around three lines of business:
Apparel – 60,000 SKU, Footwear – 25000 SKU, and Equipment -
1000 SKU.
• 90% of the business comes from 6 countries in Europe.
• The product life cycles are short.
• Uncertainty of demand is an important characteristic.
• Company was continuously improving its supply chain
management through a better integration of operations across
subsequent echelons and separate functions in the value
chain.
• As part of these efforts management decided to assign some
of their resources for improvement projects to performance
management. The integration of various local performance
indicators into a company-wide consistent system was
required.
Lohman, C., et al. (2004) “Designing a performance measurement system: A case
study”. European Journal of Operational Research, v.156, pp.267-286.

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 38
Case Study: Nike

The project objectives:

• To develop a set of high-level performance metrics


tailored to the specific business needs for use by the
senior supply chain management team, i.e. Operations,
while including existing local metrics as much as possible
and sensible.

• To design a format, i.e. a scorecard, displaying the metric


scores at the level of Nike as well as that of the business
units.

Lohman, C., et al. (2004) “Designing a performance measurement system: A case


study”. European Journal of Operational Research, v.156, pp.267-286.

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 39
Case Study: Nike
• A structure with three layers is used for displaying the information
• High Level
• Mid Level
• Lower Level
• Normalization -
• All the metrics were normalized based on a linear 0–10 scale.
• Usage -
• Senior supply chain management team : director Operations, the
functional directors of Transportation, Warehousing, and Customer
Service and the three business unit Operations directors. They will use
the scorecard on a monthly basis to facilitate review of the
organizations performance.
• General Manager Nike Europe to facilitate a quarterly review of

Operations

• Maintenance
• Monthly scorecard reviews
• Yearly redesign of the scorecard and its contents when launching new
business plans Lohman, C., et al. (2004) “Designing a performance measurement system: A case
study”. European Journal of Operational Research, v.156, pp.267-286.

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 40
Case Study: Nike
Developing and Embedding Performance Measurement System
• Defined a metrics dictionary with some 100 fixed metrics combined with an
overview of existing initiatives, while being much more flexible regarding
the structure.
• Used a hierarchical structure within each cluster, thereby using so-called
‘‘engineered indicators’’ i.e. metrics based on two or more lower-level
metrics.
• Brought together all people working on parallel initiatives in the field of PM
within the organization.
• Experimented with different ways of clustering the metrics and presenting
the scorecards.
• Gathered the data required for actually measuring and reporting all metrics
in a reliable way.
• One person of the cross-functional initiatives group managed the

performance reporting process.

• The PM manager has authority to make changes in the definition, to make


sure that metrics remain consistent between various areas and with the
global scorecard. Lohman, C., et al. (2004) “Designing a performance measurement system: A case
study”. European Journal of Operational Research, v.156, pp.267-286.

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 41
Case Study: Air Force Sustainment
Decline in Mission Capability Rates

“Material condition of an aircraft indicating it can perform at least


one and potentially all of its designated missions. Mission-capable
is further defined as the sum of full mission-capable and partial
mission-capable. Also called MC”. - DOD Definition of Mission Capability

Fig hte rs B o mbe rs S pe c ia l M is s io n


95

90

85

80
SOURCE: Compiled from data in
National Research Council, Aging
75 Avionics in Military Aircraft
(Washington, DC: National
70
Academy Press, 2001)

65

60

55

50

82

85

88

91

94

97

Year
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 42
Air Force Sustainment System
Component
Fails BASE-LEVEL No
Base Repair (Flightline (NRTS ) &
Mantenance) Possib le
MICA P
Component A ircraf t Repairab le
Reinsta lled CA NN this s tation?
(RTS)
Y es
Base
(RTS)
Supply Y es or
Piec e No &
Parts ava ilab le A WP at
Part Poss ible
at base? Base
CA NN MICA P Non-
Serv icable Base Traf f ic Mgmt Condemned
Unit Carcas s

TRANSPORTATION
Express Carrier

Carcass ALC /DEPOT-LEVEL


Order to w ithout Serv icable Part Re-Enters
CSI v ia Backorder Susta inment Pipeline
SBSS Of f s ite Depot
Location TMO
CRI
SBSS-
Job
Initiated Contract How to Routed
Order Out Repa ir? Repair
to Base
Carcass Non-Job Parts available
w ithout Routed at De p ot?
Backorder Repair
No Y es

SMA G CSI Repaired


CSI A WP at
Reques ts New Parts at Depot
Depot
Part Issuance A rrive
Job Routed Serv iceable Parts

Non-Job Routed Serv iceable Parts

LEGEND
Information F lows: Process:
Physica l Part F lows: Status:
Ent ity: Dec ision:

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 43
Case Study: Air Force Sustainment
Initial Field-Study Observations

• Current metrics generally foster local optimization rather


than global optimization
• Depot maintenance metrics are primarily focused on financial and
related measures of performance
• Metrics conflict with delivery of best customer support

• Current metrics generally do not allow measures of


progress towards the achievement of system-wide goals
• Impossible to trace incremental impact on the warfighter of
improvement actions at local level (e.g., incremental investment in
more parts and materials)
• Informed tradeoff decisions cannot be made at system-level

• Current metrics drive the “wrong” behavior


• Big example is cannibalization (removal of a serviceable component
from one end-item waiting for parts to repair another)
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 44
Regression Model:
Mission Capability Rate
Effect of key strategic metrics and action variables (below) on the
F-16 fully-mission-capable (FMC) rate

Fully-mission- Utilization rate 0.08


capable (FMC) rate Flying scheduling effectiveness 0.14
as a function of:
Break rate -0.06
Î
Cannibalization rate -0.15
Adjusted R2 = 0.47 Repeat discrepancy 0.03
Unscheduled maintenance -0.12
Temperature (control variable) -0.09
Precipitation (control variable) 0.05
Active duty “dummy”* variable 1.36
National Guard “dummy”* variable -0.32

* “Dummy” variable captures the impact of special characteristics peculiar to each user group.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 45
Complex Causal Loop Structure
Representation
Aircraft Utilization Rate
-
+
+

B
R
+
Sustainment Costs
+
-
- R
B

R -
R
-
Fix Rate -
Total Mission Capable Rate Cannibalization Rate + Break Rates
- + -
R R
R = Reinforcing loop + -
B = Balancing loop Flying Scheduling Effectiveness
+/- = Sign of coefficient
R
width = magnitude

Active Duty Mission Reserve Mission Guard Mission

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 46
Research Findings
“Metrics Thermostat” research reveals a far more complex metrics structure
than indicated by top-down hierarchical metrics structure currently being
used
• AFMC Metrics Handbook: Presents hierarchical metrics structure (command;
mission element board (MEB); center-process metrics)
• US Air Force Logistics Transformation Team “Balanced Scorecard” approach
Complex feedback loops show system behavioral dynamics not otherwise well
understood
• Increased fix rates lead to higher aircraft utilization, higher FMC rates, and increased
flying effectiveness
• But this also results in increased break rates, leading to lower fix rates by tapping
limited resources, and therefore lowering FMC rates
• Model quantifies net effects of complex interactions
Model provides new insights into the prevalence and effects of “cannibalization”
not available before
• Quantitative evidence provides new shows that cannibalization at field level has net
negative effect on the overall F-16 mission capability rate
• Shows how cannibalization (an emergent system property) is driving “wrong”
behavior

ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 47

You might also like