Drive revenue growth through data

IN COLLABORATION WITH PRODUCT LEADERS FROM

 Created by Product School and Heap

©2020 Product School Inc.

productschool.com

Design and layout: Andrea Gávalos.

Editors: Nathan Thomas, Ellen
Merriweather Simmonds, Adrianna
Berring, Sama Masa'deh, Zaina Hmoud

All rights reserved.

INDEX
4  Knowledge is Product

6  How to Make Product Analytics Useless

12  Product Analytics Made Simple

18  Discover a Product Focused Framework For Measuring Success

24  How YouTube is Preparing to Onboard One Billion More Users

30  How to Make the Transition to Product-Led Growth

36  Product Strategy: Your Competition Is NOT Who You Think It Is

42  When Losing a User Gets You Closer to Your Company Mission

48  How to Build Data Driven Roadmaps

54  How Media Companies Are Managing the Digital Transformation

60   How to Generate Insights and Influence Products

INTRODUCTION

KNOWLEDGE
IS PRODUCT

CARLOS GONZALEZ DE
VILLAUMBROSIA
Founder & CEO

More about Carlos

Knowledge is Product

The secret is out: Data is the critical ingredient in Product Management

success. But the way data is talked about, you could be forgiven for thinking it
was some kind of magic powder: Sprinkle some data about, and all of a sud-
den you have a product that is loved by millions.

The world just doesn’t work like that. Data alone won’t get you far. What you
need is product analytics. This is the structured process through which you
not only acquire data, but also use it to extract meaningful insights to guide
your decision making.

The Product Analytics Economy looks at how Product Managers today are
using product data for innovation, iteration and inspiration. With ample in-
sights provided by Heap, a leading digital experience insights provider, you’ll
learn how companies on the cutting edge make data work for them.

Inside, you’ll find insight from Nike, YouTube, Heap, Disney, Waze, Pipe-
drive, Bumble & Badoo and more. You’ll explore real life examples of how
leading firms are harnessing the potential of data to facilitate the transition
to Product-Led Growth, and find yourself liberated from some of the many
misconceptions that surround the use and application of data today.

Data is at the center of everything we Product Managers do, so let’s learn how
to make the most of it! Knowledge is power, but only when it is applied.

5
CHAPTER 01

HOW TO
MAKE
PRODUCT
ANALYTICS
USELESS

Only Fools Confuse the Action With Tools!

6
 MATIN MOVASSATE
Founder & Chairman

More about Matin

7
PRODUCT ANALYTICS IS
USELESS
Editor: Nathan Thomas

Matin Movassate is the co-founder of Heap, a well known data analytics

platform. So why is he interested in telling us how to make Product Analytics
useless? Matin’s thesis can be summarized in one phrase:

The Tool Is Not the Action

To illustrate this point, imagine that January 1 rolls around, and you decide,
like many others, to start exercising. But first, you need to research and Matin
Movassate is the co-founder of Heap, a well known data analytics platform.
So why is he interested in telling us how to make Product Analytics useless?
Matin’s thesis can be summarized in one phrase:

The Tool Is Not the Action

To illustrate this point, imagine that January 1 rolls around, and you decide,
like many others, to start exercising. But first, you need to research and join
an appropriate gym. This takes some time and research. Then you need the
right outfit to work out in., Ok, after a couple of weeks you’ve done that. Now,
you need to determine what kind of exercise will best suit your goals.… and by
now weeks have passed, it’s already mid-February, and your motivation is
not what it was! Best just wait until next year, right?

This metaphor demonstrates the common fallacy that having the thing is the
same as doing the thing. This is not true. And while we all know that this is
not true, we often don’t act accordingly. In the Product world, argues Matin
Movassate, data companies pitch that merely having data is the key to ma-

8
 CHAPTER 01

king a better product. But, just having the data is not the same as using it to
make better decisions.

To not be useless, Product Analytics should connect the dots between the
data you have and the actual decisions you need to make to deliver something
amazing for your users.

So, how does he recommend doing this? Three key pieces of advice

Find the “Unknown Unknowns” Heap’s data science team recently

analyzed hundreds of funnels, and found that 84% of funnels deliver
incomplete or misleading information. 84%! The majority of these were
user behaviors that funnels couldn’t see: alternate paths to conversion
(missed by 63% of funnels), unseen events (59% of funnels omitted
events that correlated with lower conversion), and other unknown unk-
nowns. If you’re only tracking the “happy paths” (the paths that success-
ful users take) you’re missing all the moments of friction you can’t see.
And it’s those that cause real problems!

"There is a disproportionate payoff from the unknown, since you typically have
little to lose and plenty to gain." Nicholas Nasim Taleb, The Black Swan.

Search for impact. Ask a hundred product leaders what keeps them up
at night, and 99 will say “prioritization.” (The remaining leader is too
oblivious to know.) How to avoid prioritizing the wrong things? The main
thing is to set baseline KPIs for all parts of the product, both individual
user flows and larger adoption and engagement metrics. The best way
to set these baseline KPIs is to gather reliable usage data (not unreliable
data - see point #1!), then ruthlessly track engagement numbers against
business goals. Be explicit about what you’re not prioritizing and why.
Then submit the whole prioritization structure to the next point...

Hypothesize and iterate. The best way to figure out what to do is to learn.
The best way to learn is to make hypotheses and roll out an experiment.

9
 This process forces you to measure effectiveness against success criteria,
and adopt an experimental mindset. When every new initiative is unders-
tood as an opportunity to learn, then it becomes easier to make incremen-
tal changes because you understand that not all changes will be “success-
ful” - but they will all lead to important lessons that, over time, will allow
you to move your product in the right direction. This attitude is just as
important for fixing the product directly as it is for deciding what areas in
the product to focus on.

“The value of a hypothesis is how much signal it gives you, not necessarily
whether it’s right or wrong.” - Matin Movassate

No product is ever perfect. All Product Managers strive to improve their pro-
ducts over time. Data is essential for this, but it won’t get you there alone. To
improve, use data combined with a smart, tested strategy that will, over time,
get you, your team and your product moving in the right direction. Tools ma-
tter, but only if you use them!

10
11
CHAPTER 02

PRODUCT
ANALYTICS
MADE SIM-
PLE

How to Get Started With Analytics

Without Getting Overwhelmed

12
 ENZO AVIGO
Co-founder and CEO

More about Enzo

13
PRODUCT ANALYTICS
MADE SIMPLE
Editor: Zaina Hmoud

Product Analytics may seem difficult or inaccessible to some with the satura-
tion of available data and noise in the industry. However, with a clear goal set
in place, achieving the right analytics stack and ramping up your skills can be
quickly achieved.

In this chapter Enzo Avigo, former PM at Intercom and Co-founder and CEO
at June.so, proposes a framework of simple steps to help Product Managers
finally cut the noise and uncover invaluable insights about their features or
products.

We’ll look at a four step process for getting started with Product Analytics
without getting overwhelmed, and then drill down into some of the most
useful metrics you should be keeping an eye out for right from the start.

Four Simple Principles to Get Started with Product Analytics

Right

Start lean. You need to avoid heavy lifting with data projects. A typical
journey with analytics begins by setting up some success metrics, then
tracking user behaviors, which is called instrumentation.

Usually people tend to go overboard and define too many success metrics,
and then instrument a lot of things in their product, and analyze them.
This method ends up taking too long for any progress to happen. Enzo
recommends you avoid that approach and instead take on only one project
at a time and you take it along that entire journey.

14
 CHAPTER 02

Say you have a new feature you want to release. Start with defining a suc-
cess metric, then instrument two or three behaviors related to that feature
followed by tweaking around your favorite analytics visualization tool to
lend the analysis result that you're seeking.

Set your goals. When defining your goals you need to be very pragmatic
and break things down. So a goal is typically a high level achievement that
you want to drive. It will usually look something like: I want more cus-
tomers, or I want customers to spend more. You can further break these
goals into subgoals which tend to be the starting point of the project you
select to start your analysis journey mentioned above.

Seek best practices. Make sure to utilize online resources to your benefit.
You can start by talking to people around you, to heads of product, CEOs
of companies you benchmark and get the conversation flowing.

Align on mindset. Finally, you need to make sure your team is also alig-
ned with your data analytics mindset. Enzo refers to this as the integra-
tion process and ensuring it matches your company culture. This can be
done by trying to fit this data mindset into your day to day - and make it
fun!

Now that we've seen the processes to get started, let's look into the metrics
relevant for early stage companies or for your product team:

Essential Early Metrics When Starting Out with Product

Analytics

Active users. These are key to helping you reach your product market fit.
A good way to measure that is by tracking when users are active, meaning
when they're getting the unique value from your product.

15
 Feature activation. The most granular way you can get started is to un-
derstand whether or not people engage with specific features. Track the
first steps of a feature being used and use a basic funnel to understand,
from the start of your experience, how many of the people are actually
engaging at least once with one of your specific events.

Feature retention. This is basically how long people stay around on your
product. You just need that through that in a, you know, basic line curve
analysis and understand how long, how often people like are sticking to
these features or products over time.

Acquisition cohort. Understand users based on when people got started,

and whether or not they're sticking on your product. This is a convenient
way to understand your product market fit, analyze and find out when
users were most to least effective to be able to reach out to them and try to
understand why that is.

Now you know how to get started with Product Analytics without getting
overwhelmed!

16
17
CHAPTER 03

DISCOVER
A PRO-
DUCT FO-
CUSED FRA-
MEWORK
FOR MEASU-
RING SUC-
CESS
While OKRs Are Popular, They Have Their
Limitations. Here’s an Alternative.

18
RAPHA COHEN
CPO at Waze

More about Rapha

19
DISCOVER A PRODUCT FO-
CUSED FRAMEWORK FOR
MEASURING SUCCESS
OKRs, standing for Objectives and Key Results, is a popular goal-setting fra-
mework used by major organizations throughout Silicon Valley and beyond.
While it is a useful framework for Product Managers, Rapha Cohen believes
that it also has its limitations. When misapplied and used without context, it
can actually lead Product Managers to achieve the opposite of the true defini-
tion of success that they are striving towards.

Rapha Cohen proposes an alternative framework that places renewed empha-

sis on context and worldview, helping Product Managers arrive at intelligent
and sophisticated solutions befitting a true Product Philosopher: HOSKR -
Hypothesis, Objective, Signal, Key Result.

The Limitations of an OKR Framework

No connection between your Objectives and your Key Results

No evaluation of whether or not you are capable of hitting your Key Re-
sults

No clarification on whether your Objective actually matters within the

context of your organization’s mission

Make Worldviews Into Frameworks

Putting context at the center of your Product Management approach can help
you better understand your users in a more deep, practical and profound
manner. Questions to help with this are:

20
 CHAPTER 03

What motivates people?

What changes (in their lives) with your product?

Understanding the worldview of users of Waze allowed Rapha and his team
to better reach their KPIs of acquiring more miles per session. They did this
not by focusing only on the data, but by placing this data in the context of the
user's life, and understanding other things that may have motivated them,
such as altruism or fear of rejection.

Goals - Signals - Metrics

A useful system used by Google and others when building products or fea-
tures is: Goals, Signals, Metrics, this can help you understand the Objective,
Signals, Key Results aspects of HOSKR
Goals: How do you want your users to engage with your product?

Signals: What user behavior would indicate your goal has been met?

Metrics: How can you capture data about your signals?

Hypothesis in HOSKR Equals Worldview

For a worldview to be meaningful, it needs to be adaptive. People’s world-

views are not always fixed, and not all users will have the same worldviews.
Making an effort to understand users in the context of their worldview can
allow you to begin setting metrics from the right starting point, so you are
gathering data that intelligently connects your key results and signals with
your objectives, and also relates this back to the true definition of success
that you are striving towards overall.

21
Imagine HOSKR as a Loop

Your Hypothesis allows you to determine the Signals you will observe that
inform you as to whether or not you’re achieving your Key Results on the way
to your Objective. As you observe more Signals and gather more data, you
will be able to revisit your initial hypothesis, updating it accordingly as your
company grows, your user base evolves, and you gain more insight.

Carpooling Example - OKRs VS HOSKR

Here’s how a company like Google Waze could look at carpooling if they were
using OKRs:
Observation: 65% of carpool rides are with “exes” (former user pairs)

Objective: Make the experience flawless for former user pairs

Data: Most users can’t easily find their “exes” in the app

Metrics: Retention, task success

Key Result: Increase share of rides with exes from 65% to 85%

22
 CHAPTER 03

Here’s how an HOSKR framework could be applied to the same situation to

shed more light and also identify possible flaws in OKR thinking:
Hypothesis: By allowing drivers to pre-approve incoming ride requests in
exchange for a small financial upside, we will create a better experience
for riders

Objective:
Driver centric: Fully utilize my car and save more money
Rider centric: Relax and book my commute
Business centric: We can monetize this feature

Signal: Only a fraction of engaged riders are interested. Those set rides
before getting incoming requests. When such a ride is requested, riders
flow in masses

Key Results: Driver adoption: X% of drivers set carpools Y hours ahead.

Rider engagement: Riders with such an option are Z% more likely book a
Carpool vs exes

The OKR objective to maximize riders with “exes” may, as shown by the
HOSKR framework, actually resulted in fewer users of the app as it failed
to take into account the users worldview. Instead, the pre-approval option
achieved more benefit with less downside.

Conclusion

In the above example, the OKR framework is defined as an “Objective,” the

HOSKR framework defined as a signal of failure. Interesting! We see that
starting with data and building an objective from there can be more risky
than instead starting with a worldview, and building your feature/framework
from there.

“Form a cohesive view of the interactions between users actions and the universe”

23
CHAPTER 04

HOW
YOUTUBE IS
PREPA-
RING TO
ONBOARD
ONE BI-
LLION MORE
USERS
Learn the Prioritization Secrets of
YouTube’s Head of Product.

24
 MINAL MEHTA
Head of Product at Youtube

More about Minal

25
HOW YOUTUBE IS PREPA-
RING TO OBOARD ONE
BILLION MORE USERS
If you want to understand how to scale big and scale fast, look no further than
YouTube. The video streaming platform is used by everyone you know, and
everyone they know! It has 2 billion monthly users, and you better believe
they’re prepared for the next billion.

Minal Mehta is Head of Product at YouTube, focusing on their development in

emerging markets like India and Brazil, where they already have hundreds of
millions of users and are anticipating many more. She has a wealth of insight
into how companies can be prepared to scale and prepare for an avalanche of
new users.
Minal’s first piece of advice: Prioritize!

Despite YouTube’s popularity, Minal works with a very lean team. This means
prioritization is critical. But this isn’t only true for YouTube: Many of us
now work with distributed teams, perhaps for the first time, and always are
juggling various goals and commitments. Prioritization matters now, perhaps
more than ever.

Here are some of Minal’s top tips for prioritizing effectively:

Know WHAT You’re Actually Saying “Yes” To

Understanding the actual implications of what you are really saying yes to is
the key to prioritizing effectively. Something may sound simple on the surfa-
ce, but could it lead you down a rabbit-hole that will deprive you of your time,
energy, and resources?

26
 CHAPTER 04

Know WHO You’re Saying Yes For

When you have a strong understanding of both your current users and the
new users who you anticipate will be joining your service, then intelligent
prioritization can follow naturally from knowledge of your users. If you know
who your expected users are and what matters to them, then you can make
resource allocation decisions accordingly.

Prioritizing Efficiently at Work and At Home

With many of us now working from home or “living at work” prioritization

and efficiency matter in both environments which are probably not as dis-
tinct and separate as they used to be! Here are some of Minal’s tips for both
realms (spoiler alert, the same tips apply for both!)
Create a framework for decision making

Communicate these decisions clearly: follow the rule “7 times, 7 ways”

Evaluate periodically, creating space to stay open to the emergent. Make

space to listen to data that may emerge.

Simple Rules for Complex Times

All Product Managers deal with complexity all the time. Minal Mehta is an
advocate of the Cynefin system which can help you find clarity in both simple
and complex contexts.

27
Minal’s Simple Rules for Work and Life

Detach. Minal works from 9 to 6, with a quick email download at 9pm. She
ensures work and personal phones are not on the nightstand!

Make Time for the Important: Block out 2 hours a day for non-reactive
work

Manage teams: Communicate early even when you don’t have all the
answers. Take simple one on one standing meetings. Don’t multitask! Let
people resolve issues in their own time i.e asynchronously

Final Words: Know Yourself

Everyone is different, and finding your own rhythm for prioritization and
work life balance will depend a lot on who you are, what matters to you, and
what works for you. You must know your users. You must also know yourself!

28
CHAPTER 04

29
CHAPTER 05

HOW TO
MAKE THE
TRANSITION
TO
PRODUCT-LED
GROWTH

Learn How to Overcome Challenges on

the Road to Your PLG Transformation

30
 RACHEL OBSTLER
VP of Product

More about Rachel

31
HOW TO MAKE THE TRAN-
SITION TO PRODUCT-LED
GROWTH
Editor: Nathan Thomas

Rachel Obstler is the VP of Product at Heap. Heap is not only an established

Product Analytics company, they are also one of a growing number of organi-
zations whose primary means of growth is Product-Led. Product-Led orga-
nizations are replacing Sales-Led organizations across industry verticals, as
more companies awaken to the opportunities this model provides. Here, we
look at what Product-Led Growth (PLG) is, and the challenges you may face
as you try and transition to a PLG model.

What is Product-Led Growth?

Product-Led Growth is a go-to-market model where:

Your primary interaction point with your customers is in your product

Your product is your primary support tool and primary marketing tool

Your model is scalable - your ability to acquire new customers is not limi-
ted by the number of calls your sales team can make.
If you’re thinking about making the transition to Product-Led, then Rachel
has some advice to help you navigate the challenges you will encounter along
the way:

Challenge #1: Your Team

Challenge. In a Product-Led organization, growth is everyone’s responsi-

bility. Teams used to working with “growth” assigned to one person/de-

32
 CHAPTER 05

partment may be used to thinking that growth is not a shared responsibi-

lity. But being Product-Led means that growth needs to be a responsibility
of everyone - not just product or marketing, but sales, customer success,
engineering, and more.

Example. In a Product-Led organization, a PM focused on new functiona-

lity also has to keep in mind customer adoption by looking at self-service,
scalability, and the ability for users to communicate their success outside
the app. Even though the PM does not have “growth” in their title, they
still need to factor in growth in every decision they make.

Solution: Communicate clearly to your team that growth is everyone’s

responsibility. Put together a cross-functional team that involves multi-
ple stakeholders from across the company. Make it clear that everyone’s
goal is growth, but that the primary mechanism for growth needs to be
the product. That does change responsibilities, but also makes many peo-
ple’s jobs easier! Sales, for example, then gets to spend less time pursuing
unmotivated leads.

Challenge #2: Process & Metrics

Challenge. To get a cross-functional team heading in the right direction

you need to align around growth metrics. These are different from sales
metrics. While sales metrics look at pipelines and how close someone is to
a conversion, growth metrics need to factor in a range of different signals
taken from across the entire customer journey.

Example.

Growth Metrics Sales Metrics

Trial sign-ups - Trial activation - Trial conversion Stage in sales pipeline

33
 Solution. Understand the difference between growth and sales metrics.
Determine the metrics that are important for growth, and optimize for
them. Adopt a bottom-up strategy. If your bottom-up strategy is success-
ful, sales can add value by taking the momentum higher up in the organi-
zation.

Challenge #3: Technology

Challenge. Product-Led Growth can only succeed if users can easily

experience the value of your product, and easily pay you for more value.
Many products, such as demo accounts with limited functionality, violate
a key principle of Product-Led Growth by preventing the user from easily
experiencing the value of your product.

Example. Heap helped its client, Amway, use autocapture to make their
data analysis flexible so they could iterate quickly instead of having to
wait months for tagging. By removing the friction and making the value of
analytics easily accessible, Always was able to roll out Heap’s tool to over
200 of their employees.

Solution. Remove barriers from experiencing value and making payment

in your product, and align all of your organizational technology - both in-
ternal and customer facing - around PLG goals. Structure your email cam-
paigns around their stage in customer journey, as opposed to drip cam-
paigns, and use analytics tech that gives you insights into user behavior.

34
 CHAPTER 05

Challenge #4: Friction With Sales Teams

Customers Switching away from Sales-Led to Product-Led can cause

friction with sales teams, but it doesn’t have to. Fundamentally, unders-
tand that customers are in charge, and customers generally do NOT want
to have a sales call if they don’t have to.

Example. 77% of B2B buyers want to self-serve (Source: Forrester, 2015).

Solution. Communicate self-serve, Product-Led solutions as means of

supporting your sales team. Use the data you gather to provide better
leads to sales, and demonstrate to them that once your users have realized
the value of your product, they will be more willing to get on a sales call. If
friction is high, consider offering sales teams commission on self-service
sales during the transition period.

PLG presents huge opportunities for growth. Adopt it before your competitors do!

35
CHAPTER 06

PRODUCT
STRATEGY:
YOUR COM-
PETITION IS
NOT WHO
YOU THINK
IT IS

If You Want to Acquire and Retain

Customers, You Need to Know What Your
Product is Actually Competing Against.
36
 KRISHNA PANICKER
VP of Product

More about Krishna

37
PRODUCT STRATEGY:
YOUR COMPETITION IS NOT
WHO YOU THINK IT IS
Product Managers are trained to keep a wary and respectful eye on the com-
petition. But what if your competition isn’t who you think it is? As Krishna
Panicker, VP of Product at Pipedrive, explains, sometimes truly knowing your
competition requires incorporating some unconventional thinking into your
Product Strategy.

Krishna illustrates a key “red pill moment” that changed his perspective on
competition from a Product Management perspective with a story about his
time at the early days of Skype, back in 2009.

Case Study: A Product Manager at Skype, Early 2009

This story illustrates three key points:
How to view competition

How easy it is to lose

What it takes to win

Back in 2009, Krishna was an eager PM working in the early days of Skype.
They shipped many features such as screen sharing, Skype Wifie, and video
messaging, and were looking at 50% year on year growth.

The iPhone was out but it wasn’t a big part of the Skype team’s focus. At first,
it was dismissed by many as a “bad phone.” But, of course, it would go on to
transform every aspect of how people communicate with each other.

38
 CHAPTER 06

You see, 2009 was also the year that WhatsApp launched. Right now, What-
sApp are almost at 2 billion active users! Skype, on the other hand, went
through something of a rise and fall.

The smartphone era had arrived, and Skype was going to lose badly to an app
- something it had not previously even considered as competition.

Three reasons why WhatsApp beat Skype in the smartphone era

Ready-made for the smartphone era, WhatsApp had three big advantages
that Skype did not:
Simple Interface. The instant messaging experience used to be a very di-
fferent and more complex thing! What’s made it as simple as “free SMS.”

Critical Mass. WhatsApp built up a critical mass in a rapid timeframe

by blurring the lines between sending a message and sending an invite,
creating a virtuous loop. They also partnered with telecom companies so
people would buy smartphones which had WhatsApp pre-installed.

Short Circuit Contacts. Prior to WhatsApp, you had to first sign-in to

Skype, find your friend out in the world, and then send a contact request
which your friend had to accept when online. You could only then talk
when both online. WhatsApp removed all these friction points instantly.

The guiding principle that laid this successful path for WhatsApp: “How do
we reduce time for first conversation to (almost) nothing”
Fun fact: In 2009, Skype had 1000 people to WhatsApps 55!
The Paradigm Shift: Your competition is not always who (or what) you think
it is.

“Startups mostly don’t compete against each other, they compete against no one
giving a shit” - Justin Kan, founder of Twitch.

39
Skype didn’t realize they were competing with the frictionless experience of
an app built not for computers, but for smartphones. Perhaps your real com-
petition is also different to what you think it is.
It may not be a product. You may be competing with apathy. With the status
quo. As Krisha explains, it is important to start thinking about competition in
a much broader sense.

The Principle of Minimum Differentiation

This is a trap it’s all too easy for PMs to fall into. Your “standard” competitor
introduces feature Y. Your customers ask you, “Hey, where’s feature Y.”
Soon, your boss starts asking that question also. So you build feature Y.
Meanwhile, your competitor is looking at you and asking, “Hey, where’s our
feature X!” Pretty soon, your products converge. That’s dull.

Instead of competing on the level of check-boxes, change the paradigm. It

was this level of big picture thinking that allowed Netflix to bury Blockbuster
and evolve from a DVD delivery service to an online streaming platform into,
now, a content creation company.

As the CEO of Netflix, Reed Hastings, says, they are competing against
“Drinking a bottle of wine” - Netflix views their competition as any other
way of spending your evening not on Netflix. They weren’t focused on
Blockbuster. They were focused on you.

Constantly Ask Yourself These Two Questions

“What business are we in.” (The answer won’t always be as simple as you
expect).
“Who or what are we actually competing with.”

40
THINK OF ANY
NIGHT YOU DID NOT
WATCH NETFLIX... WE
COMPETE WITH DRIN-
KING A BOTTLE OF
WINE. THAT’S A PARTI-
CULARLY TOUGH ONE.
CHAPTER 07

WHEN
LOSING A
USER GETS
YOU CLO-
SER TO
YOUR COM-
PANY MIS-
SION
Customer Acquisition Matters, But Not at
All Costs.

42
 MILES NORRIS
CPO at Bumble & Badoo

More about Miles

43
WHEN LOSING A USER GETS
YOU CLOSER TO YOUR
COMPANY MISSION
Is retention as valuable as we have been told? Perhaps a controversial ques-
tion to even ask, but, it turns out, a critical one if you’re in the relationships
business! Miles Norris is the VP of Product at MagicLab, the company behind
well-known dating and friendship apps such as Bumble, Badoo, Lumen and
Chappy.

As VP of Product, Miles and his team went on a two year journey to unders-
tand what happens when the KPIs you are given from the business team, and
the core mission of your business, are in conflict.

Let’s look at the focus of MagicLab:

Mission: Build relationships

Business goals: Healthy and growing business:

In order to achieve this: Boost MAU (Monthly Active Users)

How: Increase retention

But: Retention is in conflict with the mission!

Afterall, if someone achieves the goal of finding a relationship, they will no

longer need the app.
The MagicLab product team went through four phases to first understand
and then to begin to resolve this conflict:

44
 CHAPTER 07

Phase One: Top of the Funnel

With tens of millions of new monthly users, MagicLamp figured if they focu-
sed on early engagement, they could improve retention. They looked at what
retained users had in common:
Retained users have more than one photo, so they worked to encourage
this.

Early “Aha!” moments. So, they worked to get every user a match within
24 hours.

Results. MAU growth, but no meaningful impact on Month 1 retention. Re-

tention becomes a key KPI, with much pressure on the product team to grow
retention at all costs.

Phase Two: The First Week

The product team started to focus on how to retain users for the period after
the user immediately understands the value of the product: What are the
commonalities of users who stay engaged throughout the first week?
Profile completion - users who put more of themselves into the app stay
longer.

Customized experience - the product team added more tools for the user
to make their experience unique

Results. Activity distribution. Some users had a lot of activity, others almost
none. More MAU growth. Still no meaningful impact on M1 retention.
Retention remains a key KPI

Phase Three: Investing Users

In the quest for retention, the product team now started to focus on getting
the user the actual result they want (the core user needs): Dates and

45
relationships. Since the actual dates take place off the app, they had to find a
proxy metric for this. They settled on:
Good chats. The product team improved the in-app chat experience with
stickers, GIFs, etc. This improved the frequency of use of the chat function.

Results. MAU further increased. D1 (Day 1) and D7 (Day 7) retention

increased. Finally, early signs of impact on M1 retention.

Phase Four: The Forgotten Masses

In the fever to optimize for new users, the MagicLab product team had not
been focusing on the vast majority of longer-term users of their app? Had the
core user base been neglected?
They looked at “next period retention metrics:” who, active in this period,
is also active in the next period

If users leave, then why? They asked users for deletion reasons, and that’s
when the Aha! Moment finally came: OH - it’s because they had succee-
ded and found a relationship!

Results. Explain to stakeholders that retention cannot be a core KPI as it is

in conflict with the company mission. Instead, differentiate between “Good
churn” and “Bad churn.”

So, what are the “bad churn” reasons someone would leave their app?
They broke this into two categories:

46
 CHAPTER 07

Good Churn Vs. Bad Churn in the Relationship World

Core Learnings
After a two year journey with retention as a goal, the MagicLab product team
realized the goal was not that simple. The core retention metrics not working
and distracting the team from the leavers they pulled. It’s important to
always ask these questions:

Questions
Are your metrics correct?

Is your team focused on levers they can pull

Are your business KPIs in-line with your overall mission?

The answers will help you to:

Understand your goals and levers

Communicate them clearly to the business

Stay true to your mission, even when in conflict

Sometimes, retention really may not be the goal you’re looking for!

47
CHAPTER 08

HOW TO
BUILD DATA
DRIVEN
ROADMAPS

Discover Nike’s Five-Step Process for

Agile Product Creation

48
 SUSHMA RAO
Lead Product Manager

More about Sushma

49
HOW TO BUILD DATA
DRIVEN ROADMAPS
Editor: Adrianna Berring

Sushma Rao, Lead Product Manager at Nike, has worked in everything from
architecture to aerospace engineering, a blend of experiences that has led her
to the Product Roadmap ideology she has today.

Sushma makes her roadmaps with a combination of data, instinct, and com-
munication. She has a five-step process to creating products, couched within
an agile mindset; as she moves from one step to the next she and her team
are constantly iterating and questioning their product rationale.

Step 1: Innovation and ideation

Before you even think about opening your Product Roadmap tool of choice,
you need to make sure you have the people power to effectively problem-find
and create solutions.

To do this, you need to promote a culture of innovation by creating an envi-

ronment where ALL teammates can share their opinions without fear of re-
tribution. Creating a culture of innovation is not only a strategy for long-term
profit growth, but also a best practice for coming up with quality product
solutions. This said, while you should be open to hearing any ideas, not every
idea is possible to execute, either because of logistical/resource limitations or
because of top-down vision.

Top-down vision is an important factor to consider; with every product de-

cision you make, you also have to think about how it fits into the overall com-
pany goals. But don’t throw away any ideas immediately: if your team comes

50
 CHAPTER 08

up with an incredible solution to an unmet customer need, this is a chance to

have a conversation with leadership. As an agile Product Manager, you need
to constantly revisit vision and strategy, which sometimes means convincing
leadership of your Product Roadmap and slightly shifting the direction of the
company.

This realignment from the bottom up is possible, but needs data to back it up,
which leads us to the next step in the product creation process.

Step 2: Evaluating desirability

Once you have your idea from the ideation phase, it’s time to start collec-
ting data to determine if your solution actually addresses a customer need.
If it does, then you can again use data to conduct customer gap analyses to
identify and communicate the features that should be included in your new
product.

There are many sources to get data from, most notably:

Google Analytics from your own site

Data.gov

Annual reports of publicly traded companies

Think With Google

The Pew Research Center

If you haven’t established metrics at this point in the process, this is the time
to do it. The sooner you can define your product metrics, the better, because
they’ll inform all of your decisions moving forward. Tie metrics to strategic
goals, and if the data shows you need to adjust your strategy, do it. And if
you are losing your way in big data, narrow down the data you collect only to
those which are relevant to your industry or roadmap.

51
If you’re creating a UI-based product, for example, you’d focus on engage-
ment and usage metrics like product adoption or actions taken. But if you’re
working on a SaaS-based product, you might focus on customer acquisition
cost or lifetime value.

Step 3: Evaluating viability

After doing research, you’ve now confirmed that the product solution you
have is desirable. But before diving in and building, check in again to make
sure you’re going in the right direction.

This is where you confirm that it’s viable for your company and your team
to build this product. Does it make sense for the business? Is it doable within
your timeline and with your team? Estimate the economic value and custo-
mer lifetime value of the product or feature if you were to fully build it. Your
end-goal shouldn’t just be revenue, but it’s helpful to look at revenue as a
metric to make sure your product aligns with overall business goals.

Step 4: Evaluating feasibility

The evaluation process isn’t done yet! Because of the cyclical nature of agile
Product Management, Step 4 of creating your product is yet another iteration
of evaluating your solution.

But now we’re in the launch phase, where more complicating factors enter.
Perhaps counterintuitively, there are a lot of roadblocks during this time
because you’re finally taking action on all of your thinking and building. And
taking action means there are more external factors to take into considera-
tion, and new things to measure.

It’s no longer just the solutions team working on the product; now you have
to align with other teams like tech support or operations. This creates contin-
gencies and can muddle up your product direction.

52
At this stage, use data as a tool to help you manage priorities. Find the fra-
mework that works for you, like WSJF (weighted shortest job first) or RICE
(reach, impact, effort, confidence). Use a combination of data analysis and
gut instinct to organize your priorities into a prioritization framework, and
move forward keeping data metrics and other teams' roadmaps in mind.
make space for different individual values within the overarching values of
the company. This is why giving people pre-make solutions for big problems
seldom works. You have to let them honor their own personal values at work,
and thus allow them to grow into themselves.

Step 5: Create a feedback loop

And finally, create a feedback loop for continuous and experiential learning.
Collect data from the launch based on your metrics, and use these to improve
your product. Then measure the results of those improvements, and so on.

Nike implemented these feedback loops in a few instances, which led to pro-
ducts like Nike Live, Buy-Online Pickup In-Store, and Reserve Lockers. The
common-sense wisdom of that time was that brick-and-mortar stores were
losing popularity, so it would be a mistake to invest in in-store features. But
Nike’s feedback loop data showed them that there was in fact a high demand
for in-person experiences, but for non-traditional and lower-friction ones.

Conclusion
Metrics are essential, but be careful of blind faith in data. Think like a statis-
tician; if your results don’t make sense, remember that correlation does not
equal causality and try approaching it from a different angle. And also, me-
trics are for sharing! Involve your team for better alignment and understan-
ding of the roadmap. Create your metrics with your team, and then measure
and learn from them together to keep building better and better products.
CHAPTER 09

HOW ME-
DIA COM-
PANIES ARE
MANAGING
THE DIGITAL
TRANSFOR-
MATION

The Media Landscape is Changing Day by

Day, Here’s How Product Managers are
Finding Order in the Chaos.
54
 RACHEL BAILIN
Former Director of Product

More about Rachel

55
HOW MEDIA COMPANIES
ARE MANAGING THE
DIGITAL TRANSFORMATION
Rachel Bailin is a former Director of Product at Disney, an exciting place to
observe and participate in a dramatic shift in how individuals consume media
across the globe. While the 20th century saw innovation right the way throu-
gh from silent movies to the color films that are more familiar to us, the 21st
century has already seen even more rapid changes in how creative content is
made and consumed across the globe.

In this piece, we look at some of the opportunities and challenges presented

by the digital transformation of media, from a Product Manager’s perspective.

Digital Simplifies Distribution

In the Old Media world, distribution is highly complex involving many diffe-
rent steps and different kinds of technology:
Celluloid -> Theatre Chains -> Projector

Video Tapes -> Retailers/Rentals -> VCR

Network Broadcast -> Local Radio -> TV Antenna

Channel Feed -> Cable Providers -> Cable Box

Audio Master -> Record Store -> Home Studio

In the new media world, everything is much more simple:

Stream URL -> Any Internet Connection -> Any Device

56
 CHAPTER 09

Understanding Old Media

So-called “Old Media” entities are legacy Hollywood companies such as Sony
and 20th Century Fox who carry forward a tradition of storytelling. Los Ange-
les is itself known as a city of storytelling: There is a culture of telling stories
of human experiences, and even telling stories about telling stories - see for
example Tarantino’s “Once Upon a Time in Hollywood.”

In Product-oriented companies, Rachel asserts that most people you encoun-

ter will be tech focused. In Old Media companies, you’re likely to encounter
more creative types. Being a Product Manager requires you to “negotiate”
and find the balance between these various different personalities and stake-
holders, and treat technology and video as stakeholders in themselves.

57
Old Media Opportunities
Within the Old Media world, there are now several opportunities presented by
the overlapping fields of Product Management and tech:
Sending their existing high value content to Streaming Video on Demand
(SVoD).

Capitalizing on their unique ability to deliver on bold, creative projects.

Resisting over personalization in favor of shared experiences and

creativity.

Shift from audiences to users, focusing on Lifetime Value (LTV) rather

than one-off ad revenue.

Streamline production, taking advantage of technology to increase effi-

ciency and collaboration.

Old Media Challenges

For those in the old media world, or Product Managers considering working
in these companies, there are several current challenges to overcome:
Business Uncertainty. The rules, expectations and distribution networks
that upheld the 20th century are no longer safe bets.

Culture of Tech. Old media companies are often tech-resistant, leading to

a disconnect between the older executives and their younger users.

Conflicting Strategies. Lack of a coherent, unifying vision on how to

adapt to the new world.

“If you ask five different executives at media companies what their digital strategy
is, you’ll get 7 different answers.” - Rachel Bailin

58
 CHAPTER 09

New Tech
New tech companies are firms like Netflix and HBO who are at home in the
world of digital streaming and tend to put technology first, focusing on op-
timizing individual user experience. Like Old Media, new tech is currently at
an inflection point and is facing various challenges and opportunities which
could engage a Product Manager.

New Tech Opportunities

New tech companies have several unique advantages over their legacy
counterparts:
Global, Localized Subscriber Base. A new Netflix show can easily reach
audiences around the world.

Deep Understanding of Tech. With a focus on data and analytics, new

media companies know a lot about their users.

Best Streams. Their online streaming delivery is impeccable.

Innovation. There’s a culture of creating new experiences for users.

New Tech Challenges

Production Cultures. Tech-focused firms are less adept at generating

genuinely bold and iconic content

Cultivating Brand. Does the fact that a show is created by Netflix imme-
diately make you want to watch it?

Risk Taking. A focus on optimization can lead to less creativity and more
bland, cookie-cutter content. Viewers need to be surprised!

The media landscape is changing fast, presenting opportunities and

challenges for Product Managers in both the “old” world and the new.

59
CHAPTER 10

HOW TO
GENERA-
TE INSI-
GHTS AND
INFLUENCE
PRODUCTS

How to Join the Dots Between Data

Analytics and Product Creation

60
 MATIN MOVASSATE
Founder & Chairman

More about Matin

61
HOW TO GENERATE INSI-
GHTS AND INFLUENCE PRO-
DUCTS
Editor: Ellie Merryweather

We began this book with the provocative claim about how to make Product
Analytics useless. A bold claim indeed, especially coming from an analytics
company like Heap! But the point is this: Product data is in fact enormously
valuable, it just takes a little bit of work to make your analytics tools yield
results.

Many tool companies out there will try to sell you on the lie that simply
having this data is enough to build great products. All you have to do is hand
your teams a string of numbers and POOF out pops a world-changing
product.

So if having a great dashboard is not the goal of your analytic setup, what is?
The primary way that data produces amazing products is through the insi-
ghts it provides. By understanding what an insight is, and thinking carefully
about how you acquire them, you can work to structure your product develo-
pment processes around getting more of them.

What Is An Insight?
Everyone Talks About Insights! But What Are They… And How Can You Get More
From Your Data?
Going back to basics, Heap CEO Matin Movassate shares the definition of
‘insight’ that he’s been working on, as it applies to Product Management:

62
 CHAPTER 10

WHAT IS AN INSIGHT

An insight changes the story you tell

about your product.

An insight leads to action.

Everyone has a mental model, no matter how vague, about what their product
is, what it does, how and who uses it, etc. Over time your story should beco-
me more detailed, as you learn more about your users. It should also become
more sophisticated. For example, understanding how different segments
of your users behave differently. An insight is a piece of information that
changes your presumptions or suppositions about your product. It changes
your mental model and your story. Perhaps you’ll learn that something you
thought was easy is actually pretty hard, or that people use your product in a
slightly different way.

In tech, a piece of knowledge is pretty useless if you don’t do anything with

it. Which is why an insight is also something that leads you to action. Insights
help you to change something about your product or about how you sell it, or
how you evolve your product-market fit.

Real world example: Sur La Table found that product page views correlated
highly with purchases. So they drove usage to product pages, which increased
their overall purchase rate by 6%.

How Do You Get More Of Them?

To maximise the number of insights your data provides, you need a
combination of technology and processes. Heap have a few ideas of how to get
this combination right:

63
 1. Use tools that surface the unexpected

An insight isn’t, well, insightful, if it doesn’t tell you something new. So

if it’s insights you’re after (and it should be), you need a tool that shows
you the things you don’t expect. What happens between steps in a funnel?
What are users doing that you’re not aware of? What if the moments of
friction and opportunity lie in unexpected places?

The ideal tool should be one that shows these to you automatically, so you
don’t have to find them yourself. We call them “proactive insights,” but
you can call them “what users really do.”

2. Make Hypotheses, Test with Data

If you’re looking to change your story, you need to run experiments. You
can imagine the story you tell yourself about your product as the status
quo. As a scientist, you need to challenge the status quo with a hypothesis.

Let’s say you run a SaaS site, and you write a blog series about how to use
your new feature. Your hypothesis is that releasing this blog series will
increase your adoption rate. You can test this by seeing how many people
who read the blog went on to adopt the new feature, compared to those
who did not.

MAKE HYPOTHESES, TEST WITH DATA

Challenge Measure
assumptions and test

64
 CHAPTER 10

3. Constant Evolution, Constant Iteration

The story of the genius who wakes up one morning with their whole
product planned to perfection is nice, but we’ve never seen it in real life.
Great products are built by a series of small improvements. You chip away
at your product, making adjustments and changes. Maybe you chip away
and reveal something huge that you never anticipated.

These small changes are what allow you to gather insights in the first
place. You make hypotheses, you roll out small changes, and you see how
those changes are adopted by the market. Small changes open the door to
bigger advancements.

Real world example: YouTube started as a video site. Founders wanted to

copy ‘hot or not’ and added the option for people to upload videos. People
started using the video part more and more, even when they weren’t tr-
ying to date anyone. This insight kicked off a series of iterations that lead
YouTube to being the behemoth that it is today.

How to Make Insights and Influence Products

To conclude, gathering insights and using them to influence powerful pro-

duct decisions is a cycle built of these three ideas. These are the pillars of the
approaches needed to help teams get more insights, and build better products.

As you continue to build and iterate on your products, the future of digital
product insights is looking brighter. With new product capabilities, products
like Heap are powering automated insights and removing barriers to success.
Reducing hours spent parsing through data for product and analytics teams, a
real digital experience insights platform will empower you to spend more time
building the right product.

LEARN MORE

65
 THE
PRODUCT ANALYTICS
ECONOMY
There has never been a better time in history to build digital
products. At the same time, customers' expectations are
higher than ever before. This means that delivering an
exceptional digital experience is the only way to earn trust
and drive revenue growth.

In The Product Economy, 10 Product Leaders working at some

of the best companies in the world will show you how to:

Achieve a successful digital transformation,

Create a seamless customer experience,

Build a high-performing product team

Apply a product mindset across your entire organization.

Featuring insight from

Matin Movassate, Founder & Chairman at Heap Krishna Panicker, VP of Product at Pipedrive

Enzo Avigo, Former PM at Intercom and Miles Norris, CPO at MagicLab

Co-founder and CEO at June Rapha Cohen, CPO at Waze
Rapha Cohen, CPO at Waze Sushma Rao, Lead Product Manager at Nike

Minal Mehta, Head of Product at Youtube Rachel Bailin, former Director of Product at Disney

Rachel Obstler, VP of Product at Heap

68