Building Your Engineering Metrics Program 1

Contents

Introduction 2

Why do you need an Engineering Metrics Program? 5

Step 1 - Gain visibility Into people, process and tooling bottlenecks 7

Step 2 - Drive continuous improvement 7

Step 3 - Measuring outcome and business impact 8

Building your program

9

1. Decide your objective roadmap 10

2. Create an objective roadmap for each team: 12

3. Select persona based metrics and insights for each team 13

Metrics and insights for development managers 13

Metrics and insights for program managers 17

Metrics and insights for engineering directors 20

Metrics and insights for executives 25

4. Create an improvement process 28

5. Review your progress 26

 CHAPTER 1

Introduction
Building an Engineering Metrics Program (EMP) is a complex challenge for most engineering
organizations. On one hand there is a rising need to understand productivity with an eye
towards unlocking efficiencies and developer experience, but on the other hand there is a need
to balance this with being sensitive to measuring certain aspects of engineering metrics and
prevent the metrics from being weaponized or unfairly gamed.

In a survey of 360 leading companies with 100 engineers or more, 54% companies said they
already had some basic metrics and were looking to formalize their metrics program, 20%
said they already have a robust metrics program and 26% said they are looking to formulate a
metrics program in the near future. This document serves as a practical guide to implementing
an Engineering Metrics Program.

Building Your Engineering Metrics Program 4

 CHAPTER 2

Why do you need an

Engineering Metrics
Program?
Why do you need an
Engineering Metrics Program?
“You can’t improve what you can’t measure.” - Peter Drucker

Most engineering organizations directly or indirectly desire what is described as Engineering

Excellence. Engineering Excellence is defined as the continuous improvement process that
optimizes the workforce,processes and tooling to generate the maximum impact. So in essence
one needs to

Optimize the workforce

• Have the right people on the bus who are capable

• Have the right number of people on the project

Optimize the processes and tooling

• Have frictionless process that aims to reduce cognitive load for repetitive tasks and
give enough time for creative and high value work

But the big question is how does one optimize people, process and tooling to create outsized.
Here is a flywheel on what can be used to optimize the workforce, process and tooling to
generate.

1 2
Gain Visibility Drive Continous Improvement

3 Measure Impact and Outcomes

Building Your Engineering Metrics Program 6

Step 1 - Gain visibility into people, process
and tooling bottlenecks

People bottlenecks Process bottlenecks Tooling bottlenecks

Inadequate number of People working on low Tools that are manual

people value repetitive work and slow

People with mismatched Lack of planning and Ensuring security and

capability or output frequent context switches governance

Team reorganization Poor communication Tools that don’t work

seamlessly with each other

Step 2 - Drive continuous improvement

How do you continuously optimize these different bottlenecks

Gain visibility to make strategic decisions

Optimizing Inadequate number of people - Understand if resourcing is correct and if you need
to ask for more resources.

Optimizing people with mismatchedv capability or attitude - Understand if you have the right
folks on the bus. Assuming that everyone on the team is motivated and trustworthy and has the
right capabilities is really a mistake. Also important to measure the bottlenecks in process and
tooling to either simplifying or automating the process and better intelligent tooling.

Reduce toil by automating more

Understand what capabilities can be automated - For that it is beneficial to understand what is
the state of the art in terms of tooling and processes that can drive the automation, and create
good habits so that one can drive efficiency.

Building Your Engineering Metrics Program 7

Step 3 - Measuring outcome and business
impact

This tells us if all the changes we are driving are actually making a difference and helps us
understand what optimizations are working well and which aren’t.

Metrics and Insights are an imperative part of unlocking Engineering Excellence

For all these three steps you need engineering insights and end to end visibility. Building an
Engineering Metrics Program can really help the organizations implement these three steps.

Building Your Engineering Metrics Program 8

 CHAPTER 3

Building your program

Building your program

1. Decide your objective roadmap

It is important for organizations which objectives they should focus their efforts on. Improving
Engineering Excellence is a journey and focusing on a few objectives every quarter. If the
organization is not sure what objectives they should focus on then they can do a quick
software engineering insights baseline Implementation for a few representative teams.

The baseline implementation focuses on three pillars:

Building Your Engineering Metrics Program 10

The baseline implementation will indicate contextual improvement objectives. These objectives
can be spread over different quarters. Every team or product group can decide to create their
own objective roadmap. We recommend having at least one objective which is mandatory for
each team to focus on for that quarter. For each of the objectives, it is important to understand
the key results for that metric.

Objective Key results (metrics or insights)

Reducing lead time ( DORA metrics The key metric should reduction in lead
time

Improving quality The key metric should reduction in

customer found defects

Improving dev productivity The key metric should be to see

improvements in the scores of individuals
and teams over a period of time.

Improving planning The key metric should be to see

improvements in
• Increase in delivered vs planned story
• Decrease in scope creep
• Increase or maintain stories per sprint
• Decrease in predictability score.

Improving new feature velocity The key metric should be an increase a new
feature velocity

Optimizing Resource Allocation The key metric should be to see investment

efforts align to the allocation goals that
were set for the categories defined at the
start of the exercise.

Building Your Engineering Metrics Program 11

2. Create an objective roadmap for each team

Quarter 1 Quarter 2 Quarter 3 Quarter 4

OrgWide Improving Improving Improving Improving

quality quality quality quality

Team 1 Improving new Reducing lead Improving Improving dev

feature velocity time planning productivity

Team 2 Improving dev Improving Reducing lead Improving new

productivity planning time feature velocity

Building Your Engineering Metrics Program 12

3. Select persona based metrics and insights
for each team depending on objectives
Metrics and insights for development managers

Role Objective Metrics and Actionability Frequency Benchmarks

insights of review and ideal
ranges
Development Developer Composite scores Composite Monthly Trellis score
manager productivity can be calculated Scores Use the
from various Have out of the
tools across Conversation box default
many different with your bottom thresholds in
dimensions such 10% performers. the profile.
as speed, quality, Identify any silent Follow the
impact, leadership, top performer. color coding
collaboration. The score should
Qualitative input in only be used as
the form of surveys a smoke signal
and feedback can and conversation
also be factored starter.
into it. The score
allows a quick way
to gauge what
conversations to Coding days Coding days
have with the team Ensure devs Median
or individual. Scores commit often and coding days
are a smoke signal small is 3.1
and should be used
with sensitivity.

If using Harness
Software
Engineering Insights
you can use Trellis
Scores Feature

Coding days

Building Your Engineering Metrics Program 13

 Metrics and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
Development Developer Collaboration Collaboration Monthly Collaboration
manager Productivity review report review report review report
Make sure Unapproved PRs
nothing is getting must be zero.
merged without One reviewer
reviews. Ensure should not be
a reasonable doing more than
distribution of PR 80% of reviews
Reviews.

PR cycle time PR cycle time PR cycle time

For stuck Average PR cycle
reviews enable should be less
a Propel nudge. than 5 days
Ensure none of
the stages are
in red based
on defined
thresholds

Rework report Rework report Rework report

New rework New rework
means that code should be less
is being changed than 5% of the
within 30 days of total work.
being committed
which signals
poor quality or or
incompleteness
of requirements

Building Your Engineering Metrics Program 14

 Metrics and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
Development Improving Customer/ Customer/ Weekly for Defect rate
manager quality production production customer found should be aligned
found defect found defect defect towards agreed
rate rate upon team goal
Managers should Code coverage
make sure that and test
the overall effectiveness
trend of defects can be
is trending
downwards

Code Code coverage Code coverage

coverage Ensure that code Ideal code
coverage is coverage is 80%
increasing and as or higher
close as to 100%

Test Test case

Test effectiveness creation
effectiveness Ensure that test should be aligned
case creation is towards agreed
increasing. upon team goal

When a test case

fails it’s resulting
in a bug.

Building Your Engineering Metrics Program 15

 Metrics and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
Development Agile Hygiene score Hygiene score Every sprint Hygiene score
manager hygiene Ensure Hygiene score
requirements are should ideally be
complete before over 90%
they are getting
worked on

Ensure Tech
support is
providing steps
to reproduce
and expected
outcomes for
incidents

Scope creep Scope creep Scope creep

Ensure there is Scope creep
not a lot of scope should be less
creep stories than 20%
getting added mid
sprint

Estimation Estimation Estimation:

Ensure that Ensure it
estimation is is linear
linear. Example 3
pointers should
take less time
than 5 pointers

Building Your Engineering Metrics Program 16

Metrics and insights for development managers

Metrics and Frequency Benchmarks and

Role Objective insights Actionability of review ideal ranges
Program Improving Customer/ Customer/ Every sprint Defect rate
managers quality production production should be
found defect found defect aligned towards
rate rate: Managers agreed upon
should make team goal
sure that the
overall trend
of defects
is trending
downwards

Code coverage Code coverage Code coverage

Ensure that Ideal code
code coverage coverage is 80%
is increasing and or higher
close to 80-
100%

Test Test Test case

effectiveness effectiveness creation
Ensure that test should be
case creation is aligned towards
increasing. agreed upon
team goal
When a test
case fails it’s
resulting in a
bug.

Building Your Engineering Metrics Program 17

 Metrics and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
Program Program Hygiene score Hygiene score Every sprint Hygiene score
managers hygiene Hygiene score
Ensure all stories should ideally be
are estimated over 90%
and assigned

Ensure there
aren’t too many
large Stories in a
Sprint

Ensure
requirements are
complete before
they are getting
worked on

Ensure that there

aren’t long open
epics. Epics open
for more than 6
months are not
ideal

Ensure tech
support is
providing steps
to reproduce
and expected
outcomes for
incidents

Building Your Engineering Metrics Program 18

 Metrics and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
Program Program Scope creep Scope creep: Every sprint Scope creep
managers hygiene Ensure there is should be less
not a lot of scope than 20%
creep stories
getting added
midsprint

Estimation Estimation: Estimation:

Ensure that Ensure it is linear
estimation is
linear. Example 3
pointers should
take less time
than 5 pointers

Building Your Engineering Metrics Program 19

Metrics and insights for engineering directors

Metrics and Frequency Benchmarks and

Role Objective insights Actionability of review ideal ranges
Engineering Effort Effort Create a baseline Quarterly Link to allocation
director alignment investment understanding guide
breakdown of the effort
and trends investment Ensure the effort
investment
Communicate to trends are
upper management aligned to the
and managers about agreed upon
the what are the goals defined for
expectations and each category.
actuals

Work with EMs and

Scrum masters
to set allocation
targets for teams

Building Your Engineering Metrics Program 20

 Metrics and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
Engineering Developer Composite Composite scores Monthly Composite score
director Productivity scores Have conversation Benchmark
with your bottom the scores of
10% performers. teams and/or
Identify any silent individuals. You
top performer. can benchmark
by roles or type
of product.
Leverage this
instinct to
understand how
different teams/
individuals are
operating. Have
conversations
with them to
understand
strengths and
weaknesses. If
using Harness
Software
Engineering
Insights you use
the Trellis Scores
and use out of
the box Trellis
Profiles and
Benchmarks.

Building Your Engineering Metrics Program 21

 Metrics and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
Coding days Coding days Monthly Coding days
Ensure devs commit Median coding
often and small Days is 3.1

Collaboration Collaboration Collaboration

review report review report review report
Make sure nothing unapproved PRs
is getting merged must be zero
without reviews. One reviewer
Ensure a reasonable should not be
distribution of PR doing more than
reviews. 80% of reviews

Engineering Effort PR cycle time PR cycle time Quarterly PR cycle time

director alignment For stuck reviews Average PR cycle
enable a Propel should be less
nudge. Ensure none than 5 days
of the stages are
in red based on
defined thresholds

Rework Rework report Rework report

report New rework means New rework
that code is being should be less
changed within than 5% of the
30 days of being total work.
committed which
signals poor quality
or incompleteness
of requirements

Building Your Engineering Metrics Program 22

 Metrics and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
Engineering DORA 4 key Setup a baseline 4 Key DORA
director metrics metrics To improve lead metrics
time and MTTR, Leverage the
identify the out of box DORA
bottlenecks and definitions of
make necessary Elite, High,
process changes Medium and Low

If the change failure

rate is high, focus
on improving quality

If the reliability rate

is low, focus on
improving quality

Improving Customer/ Customer/ Weekly for Defect rate

quality production production found customer should be
found defect rate: found defect aligned towards
defect rate Managers should agreed upon
make sure that the team goal
overall trend of
defects is trending
downwards

Code Code coverage Code Code coverage

coverage Ensure that code coverage Ideal code
coverage is and Test coverage is 80%
increasing and close Effectiveness or higher
to 80-100% can be
monthly
Test effectiveness
Ensure that test
case creation is
increasing.

When a test case

fails it’s resulting in
a bug
Building Your Engineering Metrics Program 23
 Metrics
and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
Engineering Program Hygiene Hygiene score Every sprint Hygiene score
director hygiene score should ideally be
Ensure all stories over 90%
are estimated and
assigned

Ensure there
aren’t too many
large stories in a
sprint

Ensure
requirements are
complete before
they are getting
worked on

Ensure that there

aren’t long open
epics. Epics open
for more than 6
months are not
ideal

Ensure Tech
support is
providing steps
to reproduce
and expected
outcomes for
incidents

Building Your Engineering Metrics Program 24

 Metrics and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
Engineering Program Scope Scope creep: Every sprint Scope creep
director hygiene creep Ensure there is should be less
not a lot of scope than 20%
creep stories
getting added
midsprint

Estimation Estimation Estimation

Ensure that Ensure it is linear
estimation is
linear. Example 3
pointers should
take less time
than 5 pointers

Metrics and insights for executives

Metrics and Frequency Benchmarks and

Role Objective insights Actionability of review ideal ranges
Executive Effort Effort Baseline Quarterly Link to allocation
alignment investment understanding of the guide
breakdown effort investment
and trends Ensure the effort
at an org Ensure the org investment trends
level units are correctly are aligned to the
allocated agreed upon goals
defined for each
category.

Building Your Engineering Metrics Program 25

 Metrics and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
Executive Developer Composite Composite Quarterly Composite Score
productivity scores can ScoresEnsure Benchmark
be calculated that Directors and the scores of
from various EMs are staying teams and/or
tools across on top of the low individuals. You
many different performers and can benchmark
dimensions are rewarding the by roles or type of
such as speed, silent heroes product. Leverage
quality, impact, this instinct to
leadership, understand how
collaboration. different teams/
Qualitative input individuals are
in the form of operating. Have
surveys and conversations
feedback can with them to
also be factored understand
into it. The strengths and
score allows a weaknesses. If
quick way to using Harness
gauge what Software
conversations Engineering
to have with Insights you use
the team or the Trellis Scores
individual. and use out of the
Scores are a box Trellis Profiles
smoke signal and Benchmarks.
and should
be used with
sensitivity.

If using Harness
Software
Engineering
Insights you
can use Trellis
Scores Feature

Building Your Engineering Metrics Program 26

 Metrics and Frequency Benchmarks and
Role Objective insights Actionability of review ideal ranges
DORA 4 key metrics Review the Monthly 4 Key DORA
metrics baseline numbers metrics
for the 5 key Leverage the
metrics out of box DORA
definitions of elite,
Ensure they are high, medium and
trending in the low
right direction

Discuss and set

targets for the
teams and ensure
that they are
being met

Improving Customer/ Customer/ Weekly for Defect rate

quality production production found Customer should be aligned
found defect defect rate: found towards agreed
rate Directors and EMs defect upon team goal
should make sure
that the overall
trend of defects
is trending
downwards

To understand Ensure the Quarterly New feature

how many new new capability velocity is aligned
capabilities by is trending with the agreed
org units are according to the upon goals for
you shipping maturity of the each team
teams

Building Your Engineering Metrics Program 27

4. Create an improvement process

We recommend managers file tickets for improving their engineering excellence. That will
ensure transparency and accountability. This plans should be reviewed by every manager
monthly with the team. They should also be reviewed at the monthly Engineering Excellence
QBR at the Director Level.

You may use the following template with an example for your action plans.

The plan must have 1 What actions will you take, 2 Who is responsible, and 3 When it will be
done.

Template 1

Opportunity /
#1 challenge Resolution Responsible Due by Completed on
PR missing Jira Create link Tech lead name Jun 30, 2023
linkage

Template 2

Opportunity area PR missing Jira linkage.

Objective

Resolution (WHAT)

Owner (WHO) Tech lead name

Complete By (WHEN) 15 days or date

Verify and close {Close when the next report shows 100% compliance}

Priority Urgent

Odditional reference(s)

Building Your Engineering Metrics Program 28

5. Review your progress
It is important to meet and review the metrics and insights to drive the continuous improvement
process. Below is the meetings we recommend

Meeting Meeting ownership Frequency Action

Manager Engineering Manager Monthly Review the manager dashboard
Excellence Meeting with the teams.

Review the improvement

tickets and commitments.

Celebrate small wins

Leverage dev productivity

and trellis reports to identify
engineers you need to have
critical conversations or
understand context

Celebrate the developers who

have made improvements

Building Your Engineering Metrics Program 29

 Meeting Meeting ownership Frequency Action
Director Engineering Director and managers Monthly Review director dashboard.
Excellence Meeting Understand if all teams are
tracking to the objectives

Give directional guidance to

managers for improvement
areas

Have conversations on which

improvement areas you need
to prioritize

Executive QBR Engineering executives Quarterly Review executive

and directors dashboards

Track progress compared to

objectives.

Building Your Engineering Metrics Program 30

Conclusion
Harness Software Engineering Insights offers out of the box capability to surface visibility
across your people, process and tooling bottlenecks. Leveraging Software Engineering Insights
engineering leaders can start building their Engineering Metrics Program and drive continuous
improvement. Schedule a 30 minute demo to learn how leading organizations are unlocking
efficiency using Harness Software Engineering insights.

Building Your Engineering Metrics Program 31

 TM
The Modern Software Delivery Platform

Follow us on Contact us on

/harnessio www.harness.io

/harnessinc