May 2020

DevOps Institute
DevOps Journey SKILbook

Performance Management:
Metrics That Matter to
Measure Your DevOps Journey
Use Key Categories Of Metrics
to Ensure Progress Of DevOps

By Eveline Oehrlich, DevOps Institute

with Andi Mann, Splunk, JP Garbani, DevOps Institute
and Karen Skiles, DevOps Institute

Eveline Oehrlich Andi Mann

 Executive Summary
The goals of DevOps are to shift into high performing
teams to deliver software and service with speed and
quality.  To show progress, teams must focus on metrics
that address the both business and IT key performance
indicators. This report details what DevOps teams must
consider when developing a metrics program. 
The Digital Economy Requires
Modern Delivery Speed

According to a 2019 CIO study by KPMG and Harvey Nash, enterprises which have ad-
opted DevOps methodologies outpace and accelerate project delivery when compared to
other enterprises1. These companies belong to the digital leaders in which the executive
board views technology delivery differently than their peers. According to the research,
one key priority for digital leaders is speed up projects to accelerate their digital transfor-
mation. As part of this, their organization employs automation in software development
and maintenance and uses methodologies such as Agile and DevOps to speed up soft-
ware delivery.

Lay Out a Plan for a

Successful Metrics Program

DevOps leaders, executives and teams need to use metrics to measure the return on the
methodology. Existing measures of software success — reduced operations costs — are
still relevant and readily calculated. But to ensure the continuous leverage of DevOps,
KPIs which include business metrics must be populated. To address relevance for busi-
ness and business outcomes, measurement programs must develop metrics which relate
to the objectives of the business and convey how IT adds value. The following must be
kept in mind.

Continuously evaluate if you are measuring the right things.

The American statistician W. Edwards Deming famously said:
“In God we trust, all others must bring data.”2 To understand
where results constitute a win, it is necessary to leverage data
to objectively assess the current situation. This requires a
continuous review of existing KPIs to achieve new goals, clari-
fy the current expectations, update the definition of success.

1
Align the metrics with the business goals. Metrics such
as time-to-value with new products and/or offers, mar-
ket-share gains, customer satisfaction, growth and profit-
ability are essential business goals. These KPIs are essential
for the business to measure success and DevOps metrics
need to align and support these business KPIs.

Balance both business and IT KPIs for ongoing success.

According to a recent study commissioned by Broadcom,
where CEOs and senior business leaders were asked on
what their organization could do to better support the busi-
ness overall they stated the following: 1. To deliver consis-
tent software quality; 2. To deliver software value faster; 3.
To improve alignment of the software organization with the
business3. Effective KPIs can support these demands.

Understandable and relevant. Metrics must be understand-

able and relevant to business executives and in the language
the business understands. While mean-time-to-resolution
(MTTR), cycle times and error budgets are important to IT,
these must be translated into something the business un-
derstands. Key performance metrics such as availability of
business applications or the cost to support a business area
are solid metrics which are easily understood by the busi-
ness teams.

Connected to business outcomes. Business executives are

concerned with introducing products and services, improv-
ing customer loyalty and satisfaction, increasing gross mar-
gins, and growing market share. IT metrics must be linked
directly to these business outcomes, specifically demon-
strating how IT initiatives contributed favorably to improv-
ing these outcomes. “Key performance indicators for the
business are time-to-market with new products and/or
offers, market-share gains, customer satisfaction, and
revenue”

2
 Don’t ignore productivity of the workforce. Not all software
within an enterprise is customer focused. Employees across
every organization are using many different internal systems
to do their job. While many organizations have transformed
towards commercial off-the-shelf (COTS) solutions support-
ing their business operations with functional solutions such
as Enterprise Resource Planning (ERP) e.g. SAP or Customer
Relationship Management (CRM) e.g. Salesforce, there are still
development efforts for custom inhouse software or modifi-
cations to the COTS. No matter if custom or COTS, poor user
interfaces require more training and support and organiza-
tions must be keen to maximize the efficiency and productivi-
ty of their workforce. It is important to know that all software
has a massive impact on the customer experience as these
backend systems ensure timely responses; help deliver the
product; and facilitate customer relationships.

Prioritize and Start with Four Pillars

for Establishing DevOps Metrics

To successfully design DevOps metrics is to start with the end goal in mind. The end
goals that matter to most organizations are the improvements around culture, velocity,
quality and productivity. For each of these categories it is necessary to track how the
metrics within vary over time because this will show if you are succeeding in your efforts
to improve things. It will also enable you with early indications of any negative trends.
Second, not all metrics are key performance indicators. As the word “key” already de-
scribes these metrics should be closely aligned with your critical business goals. Select a
few key metrics which can become your KPI’s and ensure that you keep reviewing their
usefulness. To structure the KPIs we are using the culture, automation, customer experi-
ence and sharing as foundational pillars for a metrics program. Where culture and shar-
ing are more focused on outcomes and goals within the DevOps team, automation and
customer experience are focused towards the alignment with business. Your team should
choose from these pillars according to your organization’s goals and current situation.

3
Culture must be measured to understand if it accelerates
or hinders your DevOps initiative. Many of the barriers to
faster, better software delivery are caused by people and
process issues - like overworked (or underworked) teams,
skill gaps, task switching, dropped handoffs, or ‘gold-plating’.
Therefore, DevOps is often considered primarily a cultural
transformation. By measuring the cultural impact of DevOps
adoption on human activity - like collaboration, commu-
nication, productivity, job satisfaction - you will be able to
recognize, diagnose, and rectify many of the proximate root
causes of multi- and cross-team dysfunction.

Automation which increases velocity must be considered as

it affects your company’s revenue. Automation can improve
software delivery speed - by acting on code, systems, and
processes (e.g. Code scanning, service provisioning, problem
remediation) at machine pace; quality - by ensuring (even
enforcing) the use of only known-good processes; and even
security - by ensuring activities are executed only as de-
fined, only by authorized team members, and with an audit
trail for every action. By tracking the execution and impact
of automation, you will know that the ‘known good’ process-
es are acting as expected, being used properly, and helping
you to achieve key objectives.

Include metrics that drive strategic advantage through

improved customer experience and flow. Management guru
Peter Drucker (perhaps apocryphally) has noted, “you can’t
manage what you can’t measure”.4 While this is not always
true, good metrics certainly help direct work toward ac-
tivities that improve customer experience, and away from
‘thrashing’ on work that is not aligned with customer-centric
outcomes. If you only measure systems (uptime, CPU, temp,
response), you may have great stats on your tech, but no
idea if it is delivering desired customer outcomes and busi-
ness value.

4
 Sharing increases productivity which saves time and cost.
Sharing is at the heart of the cultural change that is DevOps
- sharing ‘technology’ elements like code, infrastructure,
documentation, services, tooling, data, etc.; but also shar-
ing ‘human’ elements like plans, responsibilities, resources,
processes, priorities, etc.. As teams collaborate and share,
research shows clear benefits to anything from cycle time
and code quality to corporate revenue and share price. Col-
laborative cultures are also more likely to attract and retain
the best talent. By measuring collaboration and sharing you
will better understand the human impact of your DevOps
transformation.

Establish DevOps KPIs by Populating

and Balancing the Four Key KPI Pillars

To successfully design DevOps metrics is to start with populating metrics for the topics
of to structure the KPIs we are using the culture, automation, customer experience and
sharing as foundational pillars for a metrics program. Important is to pick metrics that
motivate your teams and track them over time.

KPIs for measuring your DevOps culture. DevOps teams

are comprised of multiple types of expertise to produce
high-quality software. Individuals and teams need to work
across existing silos not only to accurately build products
but also to resolve major issues and outside threats. Col-
laboration, trust and knowledge sharing are key skills and
requirements. To shape the DevOps culture, teams must
support the changes in culture leveraging the following
three key performance indicators key metrics (See Figure 1).

5
 1 Employee ‘happiness’ is a critical KPI to sustain mo-
mentum. The key metrics are employee satisfaction;
absenteeism; retention/attrition; eNPS5; on-call acti-
vations; recruitment pipeline; promotion rates; em-
ployee engagement.

2 Staff productivity shows how efficient the human

is in working in his-her job. Key metrics are commit
counts; cycle times; change lead time; defect counts;
time spent in support.

3 Continuous improvement is the ongoing improve-

ment of process, services or products. Key metrics are
error budget usage; trends for mean-time-between-
failure (MTBF), mean-time-to-detect (MTTD), mean-
time-to-acknowledge (MTTA) and mean-time-to-re-
solve (MTTR).

Figure 1:
Key Performance Indicators Culture/People​

KPIs For Metrics That Matter Metrics That Matter

Insights
Culture /People for the Business for DevOps Team

Employee How happy and satisfied Employee Happiness Process

Happiness are the team members? and Satisfaction Improvements

Staff How has the Staff Productivity Productivity

Productivity culture changed? Improvements

Continuous How productive DevOps Team

Improvements is the team? Attrition

Source: DevOps Institute​

6
KPIs of measuring effective automation are centered
around velocity, quality and flow metrics. Achieving speed
means to understand the time to complete the different
steps related to the work performed in the value chain.
The goal is to ensure faster delivery and deployment, and
should be leveraged to support a company’s critical success
factor to increase its market share. Measuring automation
has significant impacts on market share and revenue as it
improves quality while it increases code releases and with
it the release of software and services to customers. The
following are three key performance indicators that support
velocity and quality of software (See Figure 2).

1 Automation efficacy provides evidence for velocity.

Key metrics are deployment frequency; cycle times;
test executions; time in software delivery lifecycle
(SDLC) phases (e.g. build times).

2 Software quality is a no brainer. Key metrics are QA

pass/fail; deploy success; defects before/after prod;
MTBF/change failure rate; service restore times, and
test coverage.

3 Automation utilization shows process efficacy. Key

metrics are process execution; success/fail rate; man-
ual activities; audit exceptions.

7
Figure 2:
Key Performance Indicators Automation

KPIs for Insights Metrics That Matter Metrics That Matter

Automation to Gain for the Business for DevOps Team

Automation Velocity of quality releases Release Frequency for Percentage of Deployment

Efficacy available to customers?
Software Annual release rate to
Quality achieve its objectives?
Automation Does the release velocity
Utilization match business goals?
What is the average time
per release cycle?
How long does each stage
in the SLDC take?
Customer Usable Releases Pipeline Automated
with Tools
Time/Cost per Release
MTBF,
Rate of Innovation Change Failure Rate
Mean Tme to
Approve Release
Mean Tme to
Deploy a Release

What does a release cost?

Source: DevOps Institute​

KPIs focusing on customer experience to ensure winning

and retaining customers. Quality metrics are focused on
the problems and failures which occur during the different
steps in the software development and deployment value
chain. The goal should be to eliminate defects before they
happen – so before software or services are used (in pro-
duction) - therefore spend less time fixing them and ensur-
ing limited or (even better) no impact (See Figure 3).

1 Systems or application performance is an essential

health KPI. Key metrics are availability; response time;
uptime; SLO achievement; transaction/microservice
‘RED’ (Rate/Errors/Duration).

2 Fast incidents and resolution times prohibit impact on

customer experience. Key metrics are incident count/
severity; MTBF; MTTD; MTTA; MTTR, change/suc-
cess failure rate. 8
 3 Measure customer loyalty to win against the compe-
tition. Key metrics are sign-ups/opt-ins; D/MAU; cart
fulfillment/abandonment; revenue; time on site/page;
app rating; returns; attach rates.

4 KPIs for employee productivity should be developed.

There are a key set of metrics which include basic
things like transaction completion time, employee
errors due to poor navigation design and end user
experience.

5 Cycle time measures how long it takes to deliver

value to the customer. While measuring the quality
of software being used by customers and employees
is important, another important aspect is to measure
how fast we deliver value and how much value is
delivered.6 To start it is best to measure cycle time for
specific tasks, e.g. feature or bug. It involves measur-
ing the time it takes from the start, e.g. start of coding
or bug detection until released to production or issue
is resolved and code change is released.

Figure 3:
Key Performance Indicators Customer Experience

KPIs for Insights Metrics That Matter Metrics That Matter

Customer to Gain for the Business for DevOps Team
Experience

System or How is the quality of Mean-Time-to-Recover System or Application

Application Health releases improving
customer experience?
Incident and
Resolution Times How is the DevOps team
working together during
Customer Loyalty the lifecycle of a release?
and Experience
How good is the DevOps
team in sharing artifacts
to avoid problems later?
(Indicates how quickly a Performance Health
former state can be restored)
Fast Incident and
Customer Loyalty Resolution Times
Customer Experience SLO Achievement
Overall
End User Experience
Employee Productivity
within the Business Cycle Time

Source: DevOps Institute​

9
 KPIs for sharing knowledge and shared responsibilities to
drive a solid DevOps culture. As the culture should be a
sharing culture, mechanisms and metrics to promote, facili-
tate and reward sharing at multiple levels need to be put in
place so that this behavior is rewarded (See Figure 4).

1 Process standardization so that all follow the same

path. Key metrics are automation utilization; manual
exceptions; process duplication; shared process defi-
nitions)

2 Teamwork as an essential pillar for team health. Key

metrics are over/under-utilization; group chat/com-
ms; team epic/story completion; knowledge sharing;
resource pairing.

3 Tool/systems sharing to accelerate collaboration. Key

metrics are shared/overlapping tools; tool prolifera-
tion; tool longevity; skills gaps.

Figure 4:
Key Performance Indicators Sharing to Improve Efficiency

KPIs For Insights Metrics That Matter Metrics That Matter

Sharing to to Gain for the Business for DevOps Team
Improve Efficiency

Process How is the sharing Possible Automation

Standardization and knowledge exchange Skill Gaps Utilization
process working?
Teamwork Process
How is the team Adherence
Tool/System working together?
Sharing Capability Knowledge
Where are Sharing
the skill gaps?
Tool
How well are tools Utilization
aiding the flow of work?

Source: DevOps Institute​

10
What This Means
Data is the Critical Ingredient to Populate Metrics Towards Action

Solid metric programs must have data to support it. Without data there are no actionable
metrics being populated. For this we suggest leveraging the SMART (Specific; Measur-
able; Achievable (or Attainable); Relevant; and Time-bound) principle populating your
metrics with data:

Select metrics which are specific (S): Well understood metrics

must be relevant, specific to the KPI with the appropriate level
of context for relevance to those both populating the metric
and using the metric. Instead of a vague goal of “increasing
collaboration” you should create a KPI to track the specific
activity that will be most meaningful for improved collabo-
ration. This might be to increase the number of physical and
virtual collaboration meetings or the time spent in meetings for
DevOps teams.

Metrics must be measurable (M): Ensure that you select quan-

tifiable and measurable metrics. Do not just gather anything
that can be counted, because not all data is useful in evaluating
success or necessary improvements. For example, measuring
customer satisfaction of a release it is better to determine a
range of improvements to achieve for example: “improve cus-
tomer satisfaction from 3.1 to 3.4”.

Set attainable metrics which can be achieved (A): It is especial-

ly important to set realistic goals. It does not help to set goals
which cannot be achieved as that is only frustrating to team
members which is detrimental to your metrics program. Met-
rics should be inspiring for all team members as that ensures
that people are motivated to achieve or overachieve in them.

11
 Relate metrics with each other (R): As DevOps is not a one-
man shop effort, metrics must be developed in conjunction
with other team members and teams all working together to
achieve certain goals. But many of the goals are interrelated. As
an example, if DevOps teams are focusing on speed only and
the key metric is frequency of code release. However, mea-
suring speed and excluding quality of the release will have a
negative impact on the deployment and will result in more time
and effort to fix quality issues which appear in production.

Time-bound to focus on making progress (T): Metrics should

be time bound so that they are meaningful and impactful. This
should be done by developing deadlines around specific met-
rics. These deadlines help teams to focus on key goals instead
of putting them on the backburner. For example, you could
determine to eliminate a key set of specific problems in the
error-log by end-of-the week or to establish knowledge articles
to support a workaround by end-of-day.

References
1 https://www.harveynash.com/group/mediacentre/2019%20Harvey%20Nash%20KPMG%20CIO%20
Survey.pdf
2 https://medium.com/@adambreckler/in-god-we-trust-all-others-bring-data-96784d01e9be
3 https://tools.marketimpacttools.com/go/broadcom/closethegap/?lang=en-us#figure9
4 https://history-biography.com/peter-drucker/
5 https://searchhrsoftware.techtarget.com/definition/employee-Net-Promoter-Score-eNPS
6 https://dzone.com/articles/flow-metrics-software-delivery-metrics-for-busines
About the Authors

Eveline Oehrlich is Chief Research Director at DevOps Institute. She conducts research
on topics focusing on DevOps as well as Business and IT Automation. She held the po-
sition of VP and Research Director at Forrester Research, where she led and conducted
research around a variety of topics including DevOps, Digital Operational Excellence, IT
and Enterprise Service Management, Cognitive Intelligence and Application Performance
Management for 13 years. She has advised leaders and teams across small and large en-
terprises in the world on challenges and possible changes to people, process and tech-
nology. She is the author of many research papers and thought leadership pieces and a
well-known presenter and speaker within the IT industry. Eveline has more than 25 years
of experience in IT.

Andi Mann, Chief Technology Advocate at Splunk, is an accomplished digital business

executive with extensive global expertise as a strategist, technologist, innovator, mar-
keter, and communicator. For over 30 years across five continents, Andi has built success
with startups, enterprises, vendors, governments, and as a leading research analyst and
consultant.

Andi has been named to Business Insider’s Top Thought-Provoking Enterprise Tech Execs,
Huffington Post’s Top 100 Cloud Computing Experts, and many other ‘Top …’ lists. He is
co-author of two popular books, ‘Visible Ops – Private Cloud’; and ‘The Innovative CIO’.
He blogs at ‘Andi Mann – Übergeek’ (http://pleasediscuss.com/andimann), and tweets as
@AndiMann.
About DevOps Institute

DevOps Institute is dedicated to advancing the human elements of DevOps success. As a

global member-based association, DevOps Institute is the go-to learning hub connecting
IT practitioners, education partners, consultants, talent acquisition and business execu-
tives to help pave the way to support digital transformation and the New IT.

We help advance careers and support emerging practices within the DevOps community
based on a human centered SKIL Framework, consisting of Skills, Knowledge, Ideas, and
Learning. All our work, including accreditations, research, events, and continuous learning
programs – is focused on providing the “human know-how” to modernize IT and make
DevOps succeed.

Address and Other Details

Please contact
oehrliche@devopsinstitute.com
for questions about this report.