Professional Documents
Culture Documents
Save
https://joulee.medium.com/why-your-team-needs-a-weekly-metrics-review-dcc9cce7ac3c 1/8
12/8/22, 2:35 PM Why Your Team Needs a Weekly Metrics Review | by Julie Zhuo | Nov, 2022 | Medium
Open in app
AI generated prompt: epic team meeting that feels important and glorious in an oil painting style.
645
A few weeks ago in our Manifesto for the 16
Data Informed, one of the five beliefs
presented was Company-wide familiarity with metrics rather than outsourcing to ‘data
people.’
Immediately, we were pummeled with questions: Does this really matter? Is this a
realistic expectation? How can an organization achieve this? So for our next few posts,
we’ll deep dive into how to make this lofty aspiration practical.
https://joulee.medium.com/why-your-team-needs-a-weekly-metrics-review-dcc9cce7ac3c 2/8
12/8/22, 2:35 PM Why Your Team Needs a Weekly Metrics Review | by Julie Zhuo | Nov, 2022 | Medium
With data, most teams inherently understand this — that is why dashboards are built
and links are passed around and we are all reminded to “please bookmark it and check
it often.”
Unfortunately, unless your job title includes the word data, the practice of loading said
bookmark does not frequently arise to the top of your to-do list, even if you really truly
do think data is important! Thus begins the great death spiral of dashboards — because
they go unused, they become unmaintained. Because they are unmaintained, when
you finally have a need to look at them, they’re broken and useless.
This is why data-informed teams rely on practices other than sheer will to create data
familiarity. The big three are 1. weekly metrics reviews, 2. weekly insight reports, and
3. insights reviews.
In this installment, we’ll tackle one of the single most impactful practices of building a
data-informed team: the weekly metrics review.
A weekly metrics review should be short and sweet (think 5–15 minutes, typically at
the start of a regular team meeting) and led by the data person who walks the group
through the key metrics for your collective area of work (e.g. new user growth,
revenue, conversion rates, tickets resolved).
The group should examine how key metrics have progressed over the past few weeks,
ideally by looking at a series of time-series line charts. The presenter can also prepare
a few key segments to review, for example if a certain type of user, platform, or market
https://joulee.medium.com/why-your-team-needs-a-weekly-metrics-review-dcc9cce7ac3c 3/8
12/8/22, 2:35 PM Why Your Team Needs a Weekly Metrics Review | by Julie Zhuo | Nov, 2022 | Medium
is strategically important to the team, or if the team has launched something that
Open in app
impacts a particular segment (like a new feature in a test market).
It’s best to keep the meeting lightweight. Preparation should be easy, ideally no more
than 30 minutes. Many great metrics reviews simply start with screenshots of
dashboards. The data person shouldn’t have to have all the answers at their finger tips
(why did active users spike two weeks ago?). It’s fine to circle back with an answer later.
This is an unusual proposition, especially when ambitious team are (rightfully!) wary
of wasting time in meetings. After reading an article like this and trying out this style
of meeting, you may find yourself wondering: Is this meeting actually helping us
accomplish anything? It’s common for the weekly metrics review to get cancelled,
demoted to an e-mail update, or become the spawning ground for long lists of follow-
up questions just so the group feels they did something.
And yet, we argue that familiarity with the impact of the team’s work is in of itself an
important enough goal to create a synchronous cadence around. Because we work in a
team environment, having shared context across different functions is critical. The
ideal outcome of a metrics review is that each person develops a shared understanding
of the following questions:
Creative teamwork is an infinite treadmill of trial and error — come up with ideas you
Open in app
hypothesize will lead to a desired outcome, then do the work to see if you’re right.
When you take account of the results is when you learn to hone your skills.
Take an example: let’s say we spent three months shipping Feature X. We might then
have a decision to make, should we work on Xv2 next? Simply looking at X’s results in
isolation isn’t enough to answer this question; we must consider this relative to other
possibilities — what about improving features A, B or C? When we take the time to
regularly reflect on the impact of our work through metric reviews, we build towards a
richer knowledge of what really drives our business.
Put another way, a major benefit of implementing weekly metrics reviews is that
everyone on the team will improve their product thinking. For example, a designer will
be less likely to propose an important button on the bottom of the page if he knows
there’s roughly a 2x decrease in CTR compared to the top. An engineer will be more
vigilant about holding the line on performance as she implements a new feature if
she’s aware that snappy loading is a huge driver of usage.
3. What is normal?
Every business has some level of volatility and seasonality. For example, enterprise
apps are more used during weekdays, while video games are played more on
weekends. Say you expect Januarys to be worse than Decembers for your business, and
as predicted, you see a January dip. How big would the dip have to be for you to be
worried that something is wrong?
Familiarity with the normal rhythms of your business is essential to quickly identifying
when you might need to take action because something is amiss, or when there’s likely
to be an error in your measurement. For example, if a team member reports that a
recent experiment led to a 20% increase in Key Metric X, but you know X to be
extremely difficult to move, you would be suspicious and probe further. Over time, a
well-honed understanding of what is normal leads to better forecasting and future
planning.
https://joulee.medium.com/why-your-team-needs-a-weekly-metrics-review-dcc9cce7ac3c 5/8
12/8/22, 2:35 PM Why Your Team Needs a Weekly Metrics Review | by Julie Zhuo | Nov, 2022 | Medium
Open in app
Done well, the practice of weekly metric reviews creates a shared understanding of
progress, hones a team’s strategic chops, and enables faster detection of issues.
But beware one of the secret killers of metric reviews — too many follow-up questions.
A good litmus test for whether a data question is worth asking is this: What is my
estimate for the most optimistic answer, and what will I do if that’s the real answer?
Conversely, What is my estimate for the most pessimistic answer, and what will I do if that’s
the real answer? If your actions for both questions are the same, you have a question
that’s probably not worth answering.
As an example, consider the question: What is our worst-performing country, and how
much worse is it doing compared to an average country?
If your pessimistic answer is Honestly I’m not going to do anything with this information
unless it’s one of our Top 10 countries, then having someone spend a bunch of time
querying for the result may not be worth it. (A better question might be: Which Top 10
country is performing the worst this quarter?)
https://joulee.medium.com/why-your-team-needs-a-weekly-metrics-review-dcc9cce7ac3c 6/8
12/8/22, 2:35 PM Why Your Team Needs a Weekly Metrics Review | by Julie Zhuo | Nov, 2022 | Medium
If you yourself are the directly accountable individual, then propose the idea of a
Open in app
metrics review with your data partner. If both of you agree, then ask the rest of the
team to commit to trying it for at least a quarter. Tacking the practice to the start of a
pre-existing team meeting also reduces friction.
https://joulee.medium.com/why-your-team-needs-a-weekly-metrics-review-dcc9cce7ac3c 7/8
12/8/22, 2:35 PM Why Your Team Needs a Weekly Metrics Review | by Julie Zhuo | Nov, 2022 | Medium
Open in app
https://joulee.medium.com/why-your-team-needs-a-weekly-metrics-review-dcc9cce7ac3c 8/8