10 Principles to

Improve
Decision-
Making
An Insider’s View of how Netflix
Makes Hard Decisions
By Gibson Biddle Jun 23, 2022

An Insider’s View of
how Netflix Makes
Hard Decisions
In recent years, I’ve given a talk called “Wicked Hard Decisions”
which has five “What would you do?” cases that provide an insider’s
view of how Netflix made tough product and business decisions. I
learned a lot while at Netflix and even more teaching these cases.

The skills can be learned. Below I list ten principles that will
improve your odds of making great decisions. I also outline how you
can practice decision-making everyday, whether you are CEO or
have just begun your career as a product manager.

These principles are:

Frame the stakes, process, and timeline

Embrace consumer science
Obsess over customers
Employ strategy, tactics, and metrics
Make decisions quickly
Debate, decide, then do
Learning is more important than knowing
Embrace risk and shrug off failure
Keep embracing risk
Stay humble

1.Frame the stakes,

process, and timeline

Jeff Bezos gets at the first principle in his 2016 Annual Report :

“Never use a one-size-fits-all decision-making process. Many

decisions are reversible, two-way doors. Those decisions can use a
light-weight process. For those, so what if you’re wrong?”

One of the first questions to ask is: “Is this a high stakes or low
stakes decision?” If it’s a high stakes decision — like the launch of
streaming in early 2007— you’ll need a deliberate approach by a
cross-functional team composed of high judgment individuals.

But for low stakes decisions — like killing a feature used by only 2%
of our members — a small team, close to both the problem and
customers, can make a faster decision. And if they’re wrong, they
can reverse or adapt quickly.

In looking at the variety of decisions companies make, most

are reversible, so bias towards action helps you to execute and
learn quickly. Unfortunately, many teams use heavyweight
processes for all decisions.

I’m guilty of this, myself.

At Netflix, I belabored the decision to remove an early incarnation

of sub-accounts (“Profiles”). We eventually killed the feature, but
then were overwhelmed by the negative response of the 2% of our
members who used the feature. (It turns out that most of our board
members were Profile users and felt we were ruining their
marriages by forcing them to share queues.)

We reversed course and all was well. But I’ll never get back the six
months I struggled with the decision. I would have been far better
off executing quickly, and then made adjustments based on member
response.

2. Embrace consumer
science
I define “consumer science” as the search for consumer insight
using a mix of data sources:

Qualitative . Focus groups, interviews, and usability sessions

help to define the problem space, to get inside the heads of
customers, and to learn how to package and position ideas in
ways that resonate with consumers. But I don’t use these sources
to make decisions; a few folks in a focus group does not
adequately represent your total audience. Qualitative sources
help frame how customers think about the product, let you see
the product through a new customer’s eyes, and enable you to
generate lots of new ideas.
Surveys . I use this tool to understand the demographics of
customers and to see what new features or attributes they might
value. Again, I don’t use this source to predict behavior. What
customers say and what they do aren’t always the same.
Empirical or existing data . Increasingly, companies have tons
of data — both financial and behavioral — that needs to be
organized in meaningful ways. My focus here is to build a suite
of metrics that helps form hypotheses about what drives
customer and shareholder value. Later, I test these hypotheses
via A/B tests.
A/B Testing and Machine Learning . These are the most
effective tools in the consumer science toolbox. The ability to
push out a new feature to a small subset of members in an A/B
test helps build a low-risk, “let’s test it!” mentality and predict
outcomes when you release the change to all. Today, Netflix’
personalization effort relies on algorithmic innovation that
combines offline machine learning experimentation with online
A/B testing to better predict which movies members will love —
or not.

Today, world-class consumer tech companies like Facebook,

Amazon, and Netflix are quick to embrace the “let’s test it!”
mentality and have made large investments in both A/B testing and
machine learning. This makes decision-making straightforward.
These companies already have a high confidence prediction about
the expected result when they push a large-scale change to
customers.

Netflix’s recent decision to switch from a five-star rating system to

a thumbs-up/thumbs-down model was straightforward. They tested
the new approach with hundreds of thousands of new members and
discovered a 200% increase in ratings using the simpler, “thumbs
up” system. Yes, there was grumbling from existing members like
me (I hate change!), but Netflix knew via A/B test results that
members would not cancel the service because of this change (I’m
still a member!)

Most companies don’t operate at the scale of Netflix, Facebook,

Google, and Amazon, but that doesn’t mean they can’t develop
consumer insight.

With fast-growing startups, I have found focus groups outside the

Bay Area (away from Silicon Valley freaks like me) to be very
helpful. A well-executed Net Promoter Score survey can help
startups to evaluate product-market fit. And, a well-developed set of
e-staff metrics can help form hypotheses about potential drivers for
customer delight.

These four sources of data — qualitative, survey, empirical, and

A/B testing — help form a customer-focused mindset that ensures
consumers are represented in key decisions. Today, the companies
that fully embrace “consumer science” lead the pack in the
consumer tech industry.

3. Obsess over
customers

When I worked at Netflix, I defined my job using a simple model: to

delight customers in hard-to-copy, margin-enhancing ways.

At the time, I thought the job was to balance customer and

shareholder value. By stepping away and thinking about my good
and bad decisions, I’ve learned to focus even more on customer
delight. To do this, you have to be maniacally focused on delighting
your customers.

There’s a project at Netflix I worked on that gnaws at me ten years

after the fact.

During Netflix’s DVD era, I got excited about a new merchandising

algorithm that took into account the cost of each DVD we delivered
to customers. At the time, a new release DVD cost three dollars to
deliver to a customer, while a DVD that was three to twelve months-
old cost two dollars, and a title that was more than one year-old cost
only a dollar to deliver.

By implementing a cost-based algorithm, we could add millions of

dollars to our bottom line. But I got a tepid response from our CEO
about the idea. His initial fear was that we would merchandise
cheaper titles ahead of titles we knew customers would love. I
assured him that we would only “nudge” a lower-cost title when the
customer’s predicted rating for each movie was the same. We
executed the algorithm and lowered content costs by millions of
dollars — it felt like a lot of money at the time.

As I think back on the project and Reed’s lukewarm response and

then spend time watching companies like Amazon obsess over its
customers, I’ve learned to focus even more on customer delight. I
think Reed’s ambivalence reflected his worry that I was too focused
on building shareholder value and not enough on delighting
customers. In hindsight, I wish I had focused nearly exclusively on
the projects that would potentially deliver delight — bigger catalog,
streaming on all devices, better video quality, voice control,
interactive storytelling, and any number of things Netflix is likely
testing now.

My guess is that one of Netflix’ most disastrous decisions — the

decision to separate the streaming and DVD businesses via the
launch of “Qwikster” — was due in part to the failure of Netflix to
obsess over its customers. Splitting the service in two required that
customers create and manage two separate accounts, resulting in
customer outrage — the opposite of delight. (Yes, there was a price
increase, too, which was hard for customers to stomach, especially
as both the price increase and Qwikster changes were poorly
communicated.)

Jeff Bezos describes the benefits of customer obsession in a way I

more fully appreciate:

“There are many advantages to a customer-centric approach, but

here’s the big one: customers are always beautifully, wonderfully
dissatisfied, even when they report being happy and business is
great. Even when they don’t yet know it, customers want something
better, and your desire to delight customers will drive you to invent
on their behalf.”

As you work to satisfy your inherently unsatisfiable customers and

contemplate the effect your decisions will have on them, you will
find yourself relentlessly driven down the path of innovation.

4. Employ strategy,
tactics, and metrics
Do you have a well-articulated company and product strategy? If
you do, you’re in a good place to make great decisions.

Product strategies are high-level hypotheses about what

delights customers in hard-to-copy, margin-enhancing ways.
In making decisions ask, “Is this decision on strategy?” If the
answer is no, don’t do it.

In 2005, Netflix added a shipping hub in Hawaii and retention

improved as disks took one day instead of three to arrive at a
member’s mailbox. This retention improvement led to the
hypothesis that faster delivery of DVDs nationwide would
substantially improve retention. This strategy — deliver movies
“instantly” — guided lots of projects to deliver disks faster to
customers.

We measured our progress through a metric that described the

percent of times members got their first choice disk the next day in
the mail.

Our “instant” strategy, with measured progress against the metric,

fueled lots of projects — doubling the number of shipping hubs in
the US, tuning our merchandising algorithms so that only the disks
in your local shipping hub were displayed on the site, and
eventually led to the launch of streaming. Most of these decisions
were straightforward as they were very much on strategy and we
could measure progress via a precise metric.

This metric improved from 70 to 95 percent over the course of a few

years and I’m confident it led to retention gains — Netflix’s
ultimate measure of delight.

5. Make decisions
quickly
Most decisions are low stakes and reversible, so your bias should be
to make them quickly. But what about the high stakes decisions?
What’s the right pace for these decisions?

When I was a college student, I got my pilot’s license. A lot of the

flight safety training focused on high-risk, fast-paced decision-
making. You are trained to make difficult decisions quickly,
knowing you can adjust after the fact.

You also learn that doing nothing — when flying into a cloud
without an instrument rating, for instance — is rarely the right
decision.

In flying, you’re introduced to the concept of provisional decision

making process. The concept: “With the data you have right now,
what decision would you make?” Then ask yourself, “What
additional data do I need to verify this decision? How and when can
I get that data?”

In the instance of flying into a cloud, the common provisional

decision is to make a 180-degree turn. The additional data required
is more weather information about clouds at different altitudes and
potential alternate routes.

In 2007, Blockbuster engaged in an assault on our business with

their “Total Access” program.

They offered their DVD-by-mail customers unlimited disk swaps at

their stores. We debated an appropriate response — lower prices or
do nothing — but the additional piece of data we needed was, “How
many disks are Blockbuster customers watching each month, and
based on this, how long can they afford to continue the program?”
We gave ourselves a month to get this data and found it.

The answer: Given a high rate of in-store “disk swaps” by their

customers, Blockbuster could only afford to maintain the Total
Access program for nine months. Our response: double down on
streaming. By the time Blockbuster stopped “Total Access” (at nine
months!), we had a substantially better streaming service, while
Blockbuster had none.

A simple rule of thumb in decision-making: Get 70% of the data you

need, then make a decision. If you have less data than this, you’re
potentially reckless. If you have more data than this, you’re
probably wasting too much time.

6. Debate, decide,
then do

Growing up, my parents would say to me, “Good fights make good
marriages.” (They’ve been married 55 years!)

This folksy, New England saying reinforces the idea that you need
good debates to make great big decisions together. And once the
decision is made, you need to quickly align to execute. Amazon calls
this behavior, “disagree and commit.”

At Netflix, we worked to deliver overwhelming value to our

customers.

Price was a big lever. We A/B tested pricing but there were many
debates as we interpreted the results: “How to balance Wall Street’s
expectation for profit against the growth that lower prices
delivered? How much debt should we take on in order to fund future
growth?” There were many other, potential long-term impacts of
price changes that we couldn’t evaluate via A/B tests.

Every Monday morning Netflix had an executive team meeting and

I can still remember Reed, Barry McCarthy (CFO) and Leslie

Kilgore (CMO) modeling the behavior they expected of the broader

executive team. Results of A/B price tests in hand, they would
debate the more nuanced issues of pricing.

And from time to time, Reed would ask participants to flip their
point of view and argue the opposite — an exercise that encourages
not just debate, but careful listening.

A common question I get: “You had passionate debates at Netflix,

but how did you ensure everyone got on board with the decision?”
My response, “It was part of the culture.”

To this day, I can still remember phrases that were repeated to

encourage the behavior: “disagree and commit,” “decide and do,”
and the simple concept of “well-formed adults” who could switch
seamlessly from candid, passionate debate to lockstep execution as
a team.

7. Learning is more
important than
knowing
For many tough decisions there is no right or wrong choice. What is
important is learning as much as you can about your customers,
business, and industry to improve the odds in future decisions.

Netflix spent a year preparing for the launch of its DVD-by-mail

service in the UK and then a week before launch cancelled the
project due to a rumor that Amazon was launching its own DVD-by-
mail service in the U.S.

Was it the right decision? It’s impossible to know. But we learned

the importance of focus, of saying “no,” and ignoring sunk cost as
we chose to focus on our core U.S. business.

We also knew that once streaming was ubiquitous, we’d be in a

good position to expand worldwide.

8. Embrace risk and

shrug off failure

The decision Netflix made to separate its DVD-by-mail and

streaming service in 2011 was a huge mistake.

Netflix was lampooned on late night TV, lost hundreds of

thousands of members, and their market cap dropped from $18B to
$5B. It was a high stakes, poorly executed decision.

CEO Reed Hastings acknowledged the mistake. He advocated that

self-inflicted wounds were much easier to fix than competitive
damage and that if the company stayed focused on building
customer value, Netflix would regain its health. Rather than beat
themselves up for making the mistake — which discourages future
risk-taking — the team learned from its mistake, then moved on.

Today, Netflix has a market cap of $80 billion and more than 221
million members worldwide.

9. Keep embracing
risk
Today’s leading consumer internet tech companies — Google,
Amazon, Facebook, and Netflix — are all about twenty years old.

Each of these companies got to this stage by embracing risk. On the

other hand, there are dozens of companies that were a big deal on
their tenth birthday, but are a shadow of their former selves today.
Blackberry, Yahoo!, and MySpace come to mind.

I was so impressed by Facebook when they celebrated their eighth

birthday. That year they went public, acquired WhatsApp for $19
billion dollars, experimented with drones to provide ubiquitous
internet access, and made large investments in VR through their
$3B acquisition of Oculus.

Facebook understood that in order to build a great company they

needed to maintain the same level of risk-taking that established
them as a startup. Playing it safe won’t get you there.

Today, I ask companies to describe the key projects they are

engaged in. Then I ask which projects are optimizations and which
are bold bets that might provide a 10x return.

Generally, it feels good if the optimization/innovation split is 80/20.

When it’s time to make decisions about these bold bets,
acknowledge they are high risk but carry a greater prospect for
substantial reward. Lean in to the risk.

10. Stay humble

The commitment to quickly shift from decision to fast execution —
to “decide and do” — requires focus and a degree of bravado as you
navigate the unknown.

As great exec teams develop they invariably make better decisions.

In some cases, however, the team grows overconfident and makes
stupid mistakes.

In explaining his decision to separate the streaming and DVD

services via the launch of the Qwikster DVD-by-mail service, Reed
Hastings admitted, “I messed up,” then speaks of his lack of
humility:

“It is clear from the feedback over the past two months that many
members felt we lacked respect and humility in the way we
announced the separation of DVD and streaming and the price
changes. That was certainly not our intent, and I offer my sincere
apology.”

Before the Qwikster debacle, Netflix made a series of great

decisions. Two examples: they partnered with hardware
manufactures to enable “Netflix Ready” TVs worldwide then made a
bold, highly successful bet on original content through their $100M
investment in “House of Cards.”

Ithink these highly successful decisions fueled overconfidence,

however, leading to the disastrous decision to split the DVD and
streaming product in two.

Putting decision-
making principles into
practice
Making great decisions — about product, people, and business —
requires practice. But how can you practice decision-making in your
everyday job?

Ask yourself these

questions to make
decisions day-to-day

In any meeting where a decision needs to be made, ask yourself,

“What questions do I need to answer to make this decision?’ then
ask the questions!

After that, ask yourself, “What should we do?” regardless of

whether you are the decision maker. Your final step: bravely state
your opinion and work to initiate meaningful debate.

Over time, you’ll develop the necessary decision-making skills

and people will seek out your opinion. One day, you’ll find
your CEO asking, “What would you do?” You’ll know at this
point that you are on the way to making your own “wicked
hard decisions.”

Aim for 70% right, over

time

These ten decision-making principles provide a framework for

making tough decisions, but there’s no one-size-fits-all approach.

The key is thinking about the principles and identifying which

apply to your decision. In the long-term, you’ll develop more
consumer insight with each decision you make, eventually leading
to high-quality, high-velocity decision-making.

Newly-formed management teams get it right about half the time. A

world-class management team will get it right 70% of the time. And
if a team gets it right more than 70% of the time, there’s likely

something wrong: they’re not taking on enough risk.

In his 2016 annual report letter , Bezos reinforces both the

importance and challenge of the product leader’s job:

“[Being an innovative company] requires you to experiment

patiently, accept failures, plant seeds, protect saplings, and double
down when you see customer delight. A customer-obsessed culture
best creates the conditions where all of that can happen… …but
you, the product or service owner, must understand the customer,
have a vision, and love the offering. A remarkable customer
experience starts with heart, intuition, curiosity, play, guts, taste…”

Product leadership is a tough job, with a diverse set of skills

required. But with each decision you make the odds for success get
higher, given your growing knowledge of consumer behavior. Jeff
Bezos has been at this more than 25 years. You’ll get there, too.