Professional Documents
Culture Documents
12 Metrics PDF
12 Metrics PDF
October 24,2005
Overview
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 2
Why Measure?
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 3
What is performance
measurement?
• Performance measurement is the process of measuring
efficiency, effectiveness and capability, of an action or a
process or a system, against given norm or target.
• Effectiveness is a measure of doing the right job - the extent to
which stakeholder requirements are met.
• Efficiency is a measure of doing the job right - how
economically the resources are utilized when providing a given
level of stakeholder satisfaction.
• Capability is a measure of ability required to do both the job
right and right job, in the short term as well as the long term.
This can be tangible, such as, resources, technology, or
intangible, such as a corporate culture.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 4
Characteristics of good metrics
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 5
A “Good” Metric Satisfies Three
Broad Criteria
1. Strategic
• Enable strategic planning and then drive deployment of the actions
required to achieve strategic objectives
• Ensure alignment of behavior and initiatives with strategic objectives
• Focus the organization on its priorities
2. Quantitative
• Provide a clear understanding of progress toward strategic objectives
• Provide current status, rate of improvement, and probability of
achievement
• Identify performance gaps and improvement opportunities
3. Qualitative
• Be perceived as valuable by your organization and the people involved
with the metric
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 6
Metric Elements
Metric Elements
Explanation
Title
Use exact names to avoid ambiguity
Objective/purpose The relation of the metric with the organizational objectives must be clear
Scope States the areas of business or parts of the organization that are included
Target Benchmarks must be determined in order to monitor progress
Formula The exact calculation of the metric must be known
Units of measure
What is/are the unit(s) used
Frequency The frequency of recording and reporting of the metric
Data source The exact data sources involved in calculating a metric value
Adapted from Neely, A., et al. (1995a) Performance measurement system design. 15, 80.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 7
Role of Performance Measurement
• Monitoring
• Control
• Identifying and attempt to close the gap between planned target
and actual performance
• Improvement
• Identify critical improvement opportunities
• Coordination
• Information for decision making – Leading Indicators
• Internal communication across processes
• External communication with stakeholders
• Motivation
• Align Behavior and encourage transformation
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 8
Transformation to the Lean Enterprise
Value Delivery
Stakeholder Objectives
CEO C Shareholders Customer Community
Executive board U
3 7 3 S Suppliers Employees
6 Processes T
O
5 M
Lifecycle Processes
E
2 4 2
Engineering
Manufact
ring
Pu
uring
R&D
3
Sales
R & s ing
Centers of
rc
e
l es
Manufacturing
i ne
ha
2 Excellence
D
Sa
Enabling Infrastructure
Eng
1 1 1 Processes
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 9
Metrics Challenge Today
• Hierarchical organizational architectures giving
way to networked enterprise architectures
• The evolving structure and dynamics of
networked enterprises display immense
complexity
• Metrics response to such complexity has been a
disappointment:
• Hierarchical metrics mindset still continues
• Response to greater complexity has been a metrics explosion
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 12
Strategic Metrics
• ROIC (Return on
Invested Capital)
Strategy Formulation
• Economic Value Add Management Assessment
(EVA) - Competitive Intelligence
Strategy Execution
- Internal Assessment
• Inventory Turnover
Value delivery Operations
• Revenue
• Cash flow
• Market Position
• Wall Street Expectations
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 13
Tactical Metrics
• Financial Turnover
• Productivity
Value Delivery Operations
• Supply Chain
Excellence
• Safety
• Quality
• Environment Strategy Formulation Strategy Execution
• Cost/Manufacturing
Efficiency
• Delivery Value Delivery
Operations
Balanced Score Cards
• Time to market Operations Management
Financial Management
• Education and Human Resource
• Time to Hire
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 15
Value Delivery Metrics
• Stock Price
• Revenue Strategy Formulation Strategy Execution
• On time delivery
• Customer satisfaction
and loyalty Value Delivery Operations
Shareholder Value
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 16
Time lags - Performance
Management Process
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 17
Time lags - Performance
Management Process
Yearly
Quarter
Month
Day
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 18
Performance Measurement &
Management Frameworks
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 19
Balanced Scorecard/Strategy Map
Human capital
Learning
Information capital
& Growth
Culture Organization capital Leadership
Adapted from : Kaplan and Norton, (2000) “Having Trouble with your strategy? Then Map it!” Harvard Business Review Sept_Oct 2000
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 20
Primary Purposes of the
Balanced Scorecard
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 21
Desired Characteristics of Performance
Measurement Systems
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 22
Lean Enterprise Metric
Interdependencies
Process Financial
Quality Yield/Cost Delivery
Health
Customer
Value
Quality
Training
Processes
Control
Enabling
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 23
Process Control View –
Performance Measurement System
Strategy
Strategic
Strategic Objectives Decision
Making
Evaluating Plan
Stakeholders Tactical
Targets
Control
Comparison
Function
Operational
Feed back
Action
Output Input
Process
Behavior
Performance
measures
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 24
Understanding Performance Measurement
Structures – Causal Models
Illustrative
Example
Metric Cluster
Metric Set
Individual Metric
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 25
No One “Right” Set of Metrics
“…although
“…althoughthere
theremay
maybebeaapotentially
potentiallylong
longlist
listof
ofnon-financial
non-financial
indicators,
indicators,individual
individualfirms
firmshave
havetotobe
beselective
selectivebybylinking
linking
explicitly
explicitlytheir
theirchoice
choiceof
ofindicators
indicatorsto
totheir
theircorporate
corporatestrategy.”
strategy.”
R.S. Kaplan and D.P. Norton, Harvard Business Review, January-February, 75-85 (1996)
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 26
Assessing a Performance
Measurement System
• Does it clearly define what constitutes business excellence?
• Does it provide the information required to set aggressive yet
achievable strategic objectives and stretch goals?
• Does it accurately portray our progress and probability of
achieving both long-term strategic objectives and near-term
milestones?
• Does it identify the root causes of barriers?
• Does it focus the organization on the priority improvement needs?
• Does it drive the behavior and actions required to achieve the
objectives?
• Does it align work with value?
• Is it easy to use?
• Does it involve everyone?
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 27
Metrics Will Change Over an Item’s
Life Cycle
Entity Phases Attributes
Business Emerging • Cash flow
Growth • Competitive advantage
Mature • Market share
Declining • Critical Mass
Product Concept • Creative backlog
• Potential product revenue
• Cost per feature
Development • Time to market
• Performance requirements
• Predicted product quality
• Design tocost
In market • Profitability
• Market expansion rate
Phase-out • Volume impact on cost
• Inventory
• Customersupport
Core Recognition • Inventory of skills and capabilities
• Competitive advantage
Competency
Learn • Acquire knowledge
Raytheon Systems, 1998 • Cycles of learning
Practice • Use
• Apply
• Levels of use in organization
• Deployment
Expert • Teach
• Leverage advantage
• Combine and evaluate
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 28
Process and Metric Maturity Model
4
All metrics (process, results,
Support processes are integrated with organizational, geographic, etc.)
and enable core business processes Enabling Total align with strategic objectives,
to provide competitive advantage. Processes Alignment provide competitive advantage
Customer-focused process Integrated & optimize the whole
management is applied
unconsciously Metrics reinforce & leverage
3
Core activities across all core
Common process language & specs Horizontal business processes
Integrated core processes allow a Processes Alignment Local interests are subordinated
seamless flow of work across process Integrated to the good of the whole
boundaries
2
Process metrics added &
Business process management, which Core integrated with result metrics
begins & ends with the customer is
Vertical Metrics aligned between
Processes Alignment strategy & daily activities in
established, in control, and in the Managed
conscious thinking of management. core processes
1
Little or no process focus. That Metrics are ad hoc and
which exists is primarily directed Initial Initial primarily results oriented.
internally toward local operations
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 29
Raytheon Systems, 1998
Level One: Initial
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 30
Level Two: Vertical Alignment
Definition
Vertical
Verticalalignment
alignmentisisthe
theaalignment
lignment and
andre inforcement of
reinforcement ofstrateg ic
strategic
objectives
objectiveswith
withsupportive
supportivegoals
goalsand
andprogress
progressmeasures
measuresat atall
all
levels
levelsofofthe
theorganization.
organization.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 31
Level Two Example
AAcore
core process
process related
related to
to product
product development
development activities
activities
might
mightbe
be documented,
documented,be be in
incontrol
control(repeatable ), be
(repeatable), be
consistently
consistentlydeployed
deployedacross
acrossthe
the organization,
organization,and
andhave
have
measurab le improvement
measurable improvementgains.
gains. IfIfso,
so,that
thatprocess
processis
is
probably
probablyat ator
ornear
nearLevel
Level22 maturity.
maturity. IfIfthe
the metrics
metrics
indicate
indicate variations
variationsin
inthe
the process
processresults,
results,thenthenthey
theyare
are
still
stillat
atLevel
Level11because
because the
the process
processis
isnot
notin
incontrol.
control.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 32
Level Three: Horizontal Alignment
Two phases:
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 33
Level Three Example
The
Theenterprise
enterprisemay
mayhave
haveseveral
severalcore
corecustomer-re lated processes
customer-related processes
such
suchasaswinning
winningnew
newbusiness
businessand
anddeveloping
developingnew
newproducts.
products.AAlso,
lso,
the
theenterprise
enterprisemay
mayhave
havemany
manyfunctions
functionsthat
thatsupport
supportor
orexecute
execute
these
thesecore
coreprocesses.
processes.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 34
Level Four: Total Alignment
Definition
Total
Totalalignment
alignmentis isthe
thesynergistic
synergisticinteraction
interactionofofmetrics
metricsfrom
fromall
all
support
supportprocesses
processeswithwithmetrics
metricsfrom
fromall
allcore
coreprocess
processtotoreinforce
reinforce
the
thestrategy
strategyand
andto
todrive
drivebusiness
businessexcellence.
excellence.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 35
Level Four Example
At
At Leve
Levell 4,
4, the
the enterprise
enterprise begins
begins asking
asking how
how
enabling
enabling processes
processes create
create competitive
competitive
advantage
advantage for for the
the core
core customer-re lated
customer-related
processes,
processes, rather
rather than
than what
what they
they do
do to
to improve
improve
themselves.
themselves.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 36
Level Five: Optimizing
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 37
Case Study: Nike
• The study focuses on European Operations of Nike.
• The company is organized around three lines of business:
Apparel – 60,000 SKU, Footwear – 25000 SKU, and Equipment -
1000 SKU.
• 90% of the business comes from 6 countries in Europe.
• The product life cycles are short.
• Uncertainty of demand is an important characteristic.
• Company was continuously improving its supply chain
management through a better integration of operations across
subsequent echelons and separate functions in the value
chain.
• As part of these efforts management decided to assign some
of their resources for improvement projects to performance
management. The integration of various local performance
indicators into a company-wide consistent system was
required.
Lohman, C., et al. (2004) “Designing a performance measurement system: A case
study”. European Journal of Operational Research, v.156, pp.267-286.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 38
Case Study: Nike
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 39
Case Study: Nike
• A structure with three layers is used for displaying the information
• High Level
• Mid Level
• Lower Level
• Normalization -
• All the metrics were normalized based on a linear 0–10 scale.
• Usage -
• Senior supply chain management team : director Operations, the
functional directors of Transportation, Warehousing, and Customer
Service and the three business unit Operations directors. They will use
the scorecard on a monthly basis to facilitate review of the
organizations performance.
• General Manager Nike Europe to facilitate a quarterly review of
Operations
• Maintenance
• Monthly scorecard reviews
• Yearly redesign of the scorecard and its contents when launching new
business plans Lohman, C., et al. (2004) “Designing a performance measurement system: A case
study”. European Journal of Operational Research, v.156, pp.267-286.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 40
Case Study: Nike
Developing and Embedding Performance Measurement System
• Defined a metrics dictionary with some 100 fixed metrics combined with an
overview of existing initiatives, while being much more flexible regarding
the structure.
• Used a hierarchical structure within each cluster, thereby using so-called
‘‘engineered indicators’’ i.e. metrics based on two or more lower-level
metrics.
• Brought together all people working on parallel initiatives in the field of PM
within the organization.
• Experimented with different ways of clustering the metrics and presenting
the scorecards.
• Gathered the data required for actually measuring and reporting all metrics
in a reliable way.
• One person of the cross-functional initiatives group managed the
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 41
Case Study: Air Force Sustainment
Decline in Mission Capability Rates
90
85
80
SOURCE: Compiled from data in
National Research Council, Aging
75 Avionics in Military Aircraft
(Washington, DC: National
70
Academy Press, 2001)
65
60
55
50
82
85
88
91
94
97
Year
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 42
Air Force Sustainment System
Component
Fails BASE-LEVEL No
Base Repair (Flightline (NRTS ) &
Mantenance) Possib le
MICA P
Component A ircraf t Repairab le
Reinsta lled CA NN this s tation?
(RTS)
Y es
Base
(RTS)
Supply Y es or
Piec e No &
Parts ava ilab le A WP at
Part Poss ible
at base? Base
CA NN MICA P Non-
Serv icable Base Traf f ic Mgmt Condemned
Unit Carcas s
TRANSPORTATION
Express Carrier
LEGEND
Information F lows: Process:
Physica l Part F lows: Status:
Ent ity: Dec ision:
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 43
Case Study: Air Force Sustainment
Initial Field-Study Observations
* “Dummy” variable captures the impact of special characteristics peculiar to each user group.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 45
Complex Causal Loop Structure
Representation
Aircraft Utilization Rate
-
+
+
B
R
+
Sustainment Costs
+
-
- R
B
R -
R
-
Fix Rate -
Total Mission Capable Rate Cannibalization Rate + Break Rates
- + -
R R
R = Reinforcing loop + -
B = Balancing loop Flying Scheduling Effectiveness
+/- = Sign of coefficient
R
width = magnitude
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 46
Research Findings
“Metrics Thermostat” research reveals a far more complex metrics structure
than indicated by top-down hierarchical metrics structure currently being
used
• AFMC Metrics Handbook: Presents hierarchical metrics structure (command;
mission element board (MEB); center-process metrics)
• US Air Force Logistics Transformation Team “Balanced Scorecard” approach
Complex feedback loops show system behavioral dynamics not otherwise well
understood
• Increased fix rates lead to higher aircraft utilization, higher FMC rates, and increased
flying effectiveness
• But this also results in increased break rates, leading to lower fix rates by tapping
limited resources, and therefore lowering FMC rates
• Model quantifies net effects of complex interactions
Model provides new insights into the prevalence and effects of “cannibalization”
not available before
• Quantitative evidence provides new shows that cannibalization at field level has net
negative effect on the overall F-16 mission capability rate
• Shows how cannibalization (an emergent system property) is driving “wrong”
behavior
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 47