Agile Metrics for

Predicting the Future

@BattistonMattia
Let me tell you a story...

L’aquila
 What I’ve learnt

FORECASTS FORECASTING
over needs
ESTIMATES GREAT
COMMUNICATION
Communicate options
Think probabilistically,
& risks, resist bias
not deterministically
against uncertainty
 About me

● From Verona, Italy

● Eng. Manager, Agile Coach,

Software Crafter

● Sky, Tesco

● Co-Author of “Team Guide to

Metrics for Business Decisions”

Mattia Battiston
@BattistonMattia

mattia.battiston@gmail.com
Book Discount

20% Discount:
http://leanpub.com/
metricsforbusinessdecisions
/c/MATTIA20-BZXib2F
 Business Questions

How long is this going to take?

How much can we get done in XXX time?

Can we finish this by deadline XXX ?

How fast are we going?

...
Story Points?

Low
correlation
https://github.com/Skelton
Thatcher/bizmetrics-book
Why? High WIP
 Why? Queues
FUNCTIONAL WAIT FOR RELEASE WAIT FOR
NEXT DEVELOPMENT DONE
TESTING RELEASE IN TESTING RELEASE TO
[3] [4]
[3] TEST ENV. [3] [5] PROD [∞]
doing done doing done

VALUE QUEUE
(touch time) (wait time)
Why? Unexpected Events

Blockers
Bugs
Emergencies
...
The only size that matters

“SMALL
ENOUGH”
DATA & TOOLS
 Data
Enough to start (and goes a long way):
● Story start/end dates
● 5 - 11 samples

Also useful to capture:

● Story metadata (type of work, value/failure
demand, size, etc.)
● Interesting events (helps interpret data)
● Transitions of story between states (to
analyse flow)
Data
 Tools?
Spreadsheets (start with this):
● https://github.com/SkeltonThatcher/bizmetrics-book
● http://bit.ly/SimResources (Troy Magennis)

Some Tools:
● https://actionableagile.com/ (Jira, CSV, Trello)
● http://www.senseadapt.com/ (Jira)
● https://marketplace.visualstudio.com/items?itemNam
e=agile-extensions.flowviz (Azure DevOps)
● https://getcorrello.com/ (Trello)
LEAD TIME
Start/End
Start/End
Lead Time Distribution
 Q: How Long?

Q: How long will this story take?

Q: How Long?
 Q: How Long?

Q: How long will this story take?

A: We are 80% confident that it won’t

take longer than 9 days
It might take just 4, but there is only a
50% chance so I wouldn’t make any
promises on that
In the worst case scenario we might
need 10 days, but there’s only 10%
chance that this might happen
Q: Will it be ready in time?
Q: The next scheduled release is in 4
days. Will this story be ready in time?
If it will, I need to train my staff on this
new feature
Q: Will it be ready in time?
Q: Will it be ready in time?
Q: The next scheduled release is in 4
days. Will this story be ready in time?
If it will, I need to train my staff on this
new feature

A: We’re only 50% confident that we can

make it

Nevermind then, let’s wait for the next

release. Thanks for the honesty
Q: When should we start?
Q: The other team needs us to play this
story to give them a new test
environment. When should we start it?

A: When do they need it by?

3 weeks time
Q: When should we start?
Q: When should we start?
Q: The other team needs us to play this
story to give them a new test
environment. When should we start it?

A: When do they need it by?

3 weeks time

A: We can start it as late as 9 days before

and be 90% confident we can finish it in
time. Let’s not interrupt what we’re doing.
STORY HEALTH
Q: What shall we work on?
Q: *at standup* What should we give
priority to today?
 Story Health

Days Story
Percentile
in progress Health
0% - 50% <4
50% - 80% 4-9
80% - 90% 9 - 10
90% - 100% > 10

Based on lead time distribution

Story Health
Q: What shall we work on?
Q: *at standup* What should we give
priority to today?

A: The oldest story. Let’s swarm on it and

get it done
THROUGHPUT
Throughput
Throughput
Throughput Stats
 Useful spreadsheet at:
https://github.com/SkeltonThatcher/bizmetrics-book
Q: How much can we do?
Q: How many stories can we
complete in the next Sprint?
Q: How much can we do?
Q: How much can we do?
Q: How many stories can we
complete in the next Sprint?

A: We’re 80% confident that we can

complete at least 5 stories. We might get
up to 8, but there’s only 50% chance
Rolling Wave Forecasting
Q: Can we do this much?
Q: My PO wants us to complete 10
stories in the next sprint, can we do it?
Q: Can we do this much?
Q: Can we do this much?
Q: My PO wants us to complete 10
stories in the next sprint, can we do it?

A: We only have 25% confidence. Are we

willing to take such a high risk? What is the
impact of being wrong?
FORECASTING
 Q: How Long?

Q: How long will this project/feature take?

Probabilistic Forecasting
● Predict the likelihood of uncertain events
● Monte Carlo simulation: use our historical data to
simulate what might happen in the future
● Result = list of possible outcomes + probability of
that particular outcome to become reality
 Useful spreadsheet at:
https://github.com/SkeltonThatcher/bizmetrics-book
Input
Monte Carlo - Mechanics

Randomly extracted from

past throughput
Monte Carlo - Mechanics

After 2 sprints, we simulate

to have completed 8 stories
Monte Carlo - Mechanics

In this simulation, after 5

sprints we have completed
all stories
Monte Carlo - Mechanics
Communicating the Output
 Q: How Long?

Q: How long will this project/feature take?

A: We have about 50% chance of

completing the work in 4 sprints, but we
are 85% confident that it won’t take longer
than 5.
In the worst case scenario we might need 6,
but there’s only 15% chance that this might
happen. What would be the impact?
We’ll review our forecast after each sprint
Continuous Forecasting
Continuous Forecasting
Continuous Forecasting
Continuous Forecasting
Continuous Forecasting
Continuous Forecasting
Continuous Forecasting
 Q: What can I get?

Q: What can I get in the next 4 sprints?

Q: What can I get?
 Q: What can I get?

Q: What can I get in the next 4 sprints?

A: We are 80% confident that we can

complete at least 12 stories in the next 4
sprints.
Q: Will it be ready in time?

Q: Will this feature be ready by date X?

A: This feature was broken down into 10

stories. We’re 60% confident we can
complete all 10 in time.

We are 80% confident we can complete at

least 7 stories. Which ones should we give
priority to?
 ESTIMATE FORECAST
False sense of certainty Communicates uncertainty
Expressed as single number or precise date Expressed as range of outcomes with likelihood

Inaccurate More accurate

Low correlation between estimates and lead time Based on data and measurements

Unreliable when we’re too busy Accounts for usual utilisation

Even the simplest activities take forever Historical data reflects usual level of workload

Biased towards active time Accounts for queues and problems

We only think about hands-on time, ignoring queues Historical data includes queues and past problems

Biased towards expectations Reduces personal bias

We give the result that the client wants to hear Based on data instead of personal opinions

Fixed-scope, no discovery Flexible scope, allows discovery

Fixates on a particular solution Forecast N stories, not important which ones

Encourages upfront plans Encourages risk management

Often risks are ignored Highlights impact of risks

More expensive, no reviews Less expensive, reviewed often

Need to estimate each story; rarely reviewed Reviewed often with new data (e.g. every sprint)
FORECASTING:
TIPS & TRICKS
 Tips & Tricks

Use Account for

Forecast n.
relevant “Dark
stories
data Matter”

Ask Trust the

Questions people
 Debunking Myths - 1
● “I can just use the average”
No, because of “Flaw of averages”. The average can be
very different from what happens in reality, and a
prediction based on averages can therefore be highly
inaccurate.
Moreover, the average being a single number hides
away the range of possible outcomes.
● “This only works if I have a lot of data”
5 to 11 samples is enough to start.
● “This only works if all stories have the same size”
As long as we split stories as small as possible, then it
doesn’t matter if they have different size.
The size of a story has little correlation with its lead
time, which follows a known distribution.
 Debunking Myths - 2
● “I can cheat by creating lots of tiny stories”
True, but that’s actually a good side effect. Small
stories have many benefits.
● “I can’t do this, my boss wants a number”
Sometimes people will only want to hear a number. In
this case, you can pick the confidence that you want
to have and communicate that number (but be
aware that you’re missing out on very useful
discussions).
● “If we don’t estimate the team won’t have shared
understanding”
The team still has discussions to break down work
and create shared understanding. We simply don’t
worry about putting a number on stories.
GETTING
STARTED
 Getting Started
● Keep doing what you’re doing, but start collecting
data

● Next time you estimate something, also run a

forecast

● Keep updating your forecast with new data

● Review the difference between your estimate and

your forecast
 What I’ve learnt

FORECASTS FORECASTING
over needs
ESTIMATES GREAT
COMMUNICATION
Communicate options
Think probabilistically,
& risks, resist bias
not deterministically
against uncertainty
THANK YOU!
@BattistonMattia

mattia.battiston@gmail.com

really, really appreciated! Help me improve

Resources - Books
 Resources - Links
Spreadsheets
● https://github.com/SkeltonThatcher/bizmetrics-book - Mattia Battiston & Chris Young
● http://bit.ly/SimResources - Troy Magennis

Metrics
● Kanban Metrics in Practice - Mattia Battiston (video)
● Coaching in a data-driven world - Nick Brown (video)
● Actionable Metrics for Predictability - Dan Vacanti (video)
● How An Expedite Request Sunk the Titanic - Dan Vacanti (video)
● http://bit.ly/SimResources - Troy Magennis (slides)

Forecasting
● Cycle Time Analytics - Troy Magennis (video)
● What’s the story about Agile Data - Troy Magennis (video)
● Forecasting Using Data - Troy Magennis (video)
● Your Project Behaves Like a Hurricane - Dan Vacanti (video)
● #noestimates Project Planning Using Monte Carlo Simulation - Dimitar Bakardzhiev
(video)
● Forecasting Delivery, with Oranges - Dan Brown (video)
● NoEstimates game - Matt Philip (game and resources)
 Resources - Links
Story Points
● Story Points and Velocity: The Good Bits - Pawel Brodzinski (article)
● Why you might be wasting your time with Story Point estimation - Ian Carroll (article)
● Story Points Considered Harmful - Vasco Duarte (article)
● Does size matter? - Nader Talai (slides)
● How do you estimate on an Agile project? - Various authors at Thoughtworks (article)

Lead Time
● Inside a Lead Time Distribution - Alexei Zheglov (article)
● Slack Time - Pawel Brodzinski (article)
● Economic Value of Slack Time - Pawel Brodzinski (article)