23.11.07 - CNB - Experimenting - Identifying Assumptions, Testing Customer-Problem Hypotheses, Interviewing

TEK010/TEK656
Henrik Berglund
Entrepreneurship & Strategy
Chalmers University of Technology
henber@chalmers.se
www.henrikberglund.com
@khberglund
Course Structure
Identifying
Designing
Scaling
Last time
Entrepreneurship as Design
Sources of Opportunities
Jobs-To-Be-Done
Today
Startups vs Companies
Experimentation
Learning from Customers

But first…
…group formation
…group formation
Today
Startups vs Companies
Experimentation
Learning from Customers

Startups
(What We Used to Believe

What We Now Know)
What We Used to Believe
Startups are a Smaller Version
of a Large Company
What We Now Know
Startups ≠ Small companies
Startups Experiment
Companies Execute
Strategy
Start by developing a Business Plan…
…make the financial forecasts…
…then Execute
What We Now Know
Strategy
5-Year Plans
Develop and Execute the Business Plan
Why?
“No plan of operations extends
with certainty beyond the first
encounter with the enemy's main
strength”
“No business plan survives
first contact with customers”
“Everybody has a plan until they get punched in the face”
Mike Tyson
Experimenting with a Business Model
comes before
Executing a business plan
A Business Model framework
Another Business Model framework
Value proposition
Key activities Customer
relationships
Customer
Key partners segments
Visualization of
the business
model framwork
Cost structure Key Channels Revenue
resources streams
Experimentation Execution
Business Model Operating Plan +

Strategy
Hypotheses Financial Model
Process
We Built Startups by
Managing Processes
Product Management
+
StageGate/Waterfall
Traditional Development Process
Concept/ Product Dev. Alpha/Beta Launch

Business Plan Test
Has Two Implicit Assumptions
Customer Problem: known

Business Plan Test
Product Features: known iPhone 13 iPhone 14
“Reasonable” for
incremental
development
projects targeting
existing customers

Business Plan Test
Tradition – Hire Marketing

Business Plan Test
- Create MarCom - Hire PR Agency - Create Demand

Marketing Materials - Early Buzz - Launch Event
- Create Positioning - “Branding”
Tradition – Hire Sales

Business Plan Test

- Hire Sales VP - Build Sales

Sales - Hire 1st Sales Staff Organization
Tradition – Hire Business Development

Business Plan Test


Business - Hire 1st Bus Dev - Do deals for Launch

Development
Example
Online clothes retailer specialized in
fashion and sports (e.g. Adidas, Fila, Vans,
Converse, DKNY and Fred Perry).
Raised $160 million (JP Morgan, Goldman
Sachs, Bernard Arnault, Benetton + six).
Developed an innovative site with 3D,
zoom, 360⁰ rotation and virtual
mannequins, powered by lots of JavaScript
and Flash Animations.
Rapidly grew to 420 people and spent
heavily on PR pre launch.
Delayed launch repeatedly due to
technological problems.
Finally launched in 18 countries
simultaneously 8:59 November 3 1999 EST.
What happened?
40 % of visitors could not log on to the site. “Eighty-one minutes
to pay too much money
Mac users could not operate the site at all.
for a pair of shoes that I
The site was very difficult to use and full of am still going to have to
errors, often causing computers to freeze. wait a week to get?”
Extremely slow to load without broadband.
Only one in four attempts to make a
purchase worked.
Low conversion rates (0.25%).
Conversion rates did double by Christmas.
30% returns, not 10% as projected.
A “low-bandwidth version” was relaunched
within months.
But low sales + high costs => Bankruptcy.
Validated Business Model?
"Our strong investor base offers a solid
foundation for boo.com.
Validated Business Model?
"Our strong investor base offers a solid
foundation for boo.com. The fact that
such international investors have
invested in boo.com reflects the power
of our business model and the boo.com
brand."
Patrik Hedelin, Executive Chairman.

(Press Release, Nov 3 1999)
The problem – untested assumptions!
Value proposition
"Boo.com will revolutionize the Unfair Advantage
Solution First Mover,
way we shop"
E-commerce, €€ for Branding,
”It’s a completely new lifestyle
Cool modern brand, Bokus.se
proposition.".
Free shipping
Customer
Problem segments
Access to brands, “young, well-off,
Large inventory, fashion- conscious
Want to feel hip 18 to 24 year olds”
Broadband access
Will buy online
Key Metrics
CAC vs Lifetime, Revenue
Value, Cost structure Channels
Paid & Earned Streams
Return rates Operations,
Media, Virality, Online sales/full
Inventory, Logistics,
Buying online. retail price
Marketing,
The problem – untested assumptions!
• Both Customer Problems and Product Features

are hypotheses
• Emphasis on execution rather than learning
• Often leads to premature scaling
You do not know if you are wrong until you

are out of business!
Still happens today
Especially when capital is plentiful
Business Plan Test


Business - Hire First Bus Dev - Do deals for FCS

Development
What We Now Know
Process
More startups fail from
a lack of customers than from a
failure of product development
Why?
Because
• We have (and use) processes to manage

product development
• We have no established process to manage

customer development
An Inexpensive Fix
Focus on Customers and their Problems
from Day One!
How?
Product and Customer Development
Product Development
Concept Product Dev. Alpha/Beta Launch

Test
Customer Development
Customer Customer Customer Company

Discovery Validation Creation Building
Strategy Business Model

Operating Plan +
Hypotheses
Financial Model
Process Customer & Product Management

& Waterfall Development
Agile Development
Organization
Hire and Build a
Functional Organization
What We Now Know
Organization
Founders run an agile
Customer Development Team
No sales, marketing, and business

development
Founders run an agile
Customer Development Team
No sales, marketing and business

development
Strategy Business Model
Operating Plan +
Hypotheses
Financial Model
Customer Development, Product Management

Process
Agile Development Agile or Waterfall Development
Customer Functional Organization

Organization
Development Team, by Department
Founder-driven
To repeat
To repeat
More startups fail from

a lack of customers than from a
failure of product development…
… because they think startups = small companies…
…they focus on executing the plan…
Test
• Both Customer Problems and Product Features

are hypotheses
• Emphasis on execution rather than learning and
discovery
• Often leads to premature scaling
You do not know if you are wrong until you

are out of money/business
… so they scale on untested assumptions…
Unfair Advantage
Solution Value proposition First Mover,
E-commerce, ”It’s a completely new lifestyle €€ for Branding,
Cool modern brand, proposition.". Bokus.se
Free shipping
Customer
Problem segments
Access to brands, “young, well-off,
Large inventory, fashion- conscious
Want to feel hip 18 to 24 year olds”
Broadband access
Will buy online
Key Metrics
CAC vs Lifetime, Revenue
Value, Cost structure Channels
Paid & Earned Streams
Return rates Operations,
Media, Virality, Online sales/full
Inventory, Logistics,
Buying online. retail price
Marketing,
… and end up going bust.
“We have been too visionary. We

wanted everything to be perfect, and
we have not had control of costs.”
Ernst Malmsten
(BBC News, May 18 2000)
So what to do?
EXPERIMENT
Experimentation Transformation
“Experimentation describes how a

visionary founder leads subordinates,
who use distinct and unambiguous
artifacts to test explicit hypotheses, in
order to learn about—and gradually
adapt the business to—the external
environment.”
Product and Customer Development
Product Development

Test
Customer Development
Customer Customer Customer Company

Discovery Validation Creation Building
Agile Product Development
Unit of progress: a line of working code
“Product Owner” or
in-house customer
Problem: unknown Solution: unknown

Problem: known Solution: unknown
Agile Product and Customer Development

Agile Product and Customer Development
Unit of progress: validated learning about the business

Customer Development: Key Ideas
• Parallel process to Product Development (agile)
• Progress assessed not in terms of customer insights
• Emphasis on iterative learning before execution
• Must be done by small team including CEO/project leader

Customer Development Heuristics
• There are no facts inside, so get out of the building!
• Develop for the few, not for the many
• “Earlyvangelists” make your company and are smarter than you!
• Develop a minimum viable product to maximize fast learning.

• Articulate and test your BM hypotheses
(Problem & Solution (& Distribution))
• No selling; listening and learning
• Must be done by CEO/project manager
building
building
block
block
building
block building building
building block block
block
building
block
building building
block block
build
g in g building
buildin block
block block
Identifying assumptions
A) Surface all Assumptions

1. Individually
2. As a group
B) Write down Problem Hypothesis
C) Create Customer Persona
Surface your assumptions
• Brainstorm/identify hypotheses individually

• Then discuss and cluster them as a group
Examples of hypothesis to get started
I believe that my customers have a need to _____________.
This need can be solved with_________________________.
Customers already use tools like ______________________.
My initial customers will be__________________________.
The #1 value a client wants to get out of my service is _____.
The client can also get these additional benefits ___ and __.
I will acquire most users/customers through ____and _____.
I will make money by _______________________________.
Purchasing decisions are made by _____________________.
The decision to buy is influence by _____________________.
Stakeholders involved in buying decisions are ____________.
Stakeholders involved in implementing solutions are ______.
Cluster around themes
• Brainstorm/identify hypotheses individually

• Then discuss and cluster them as a group
Map them onto BM-framework
hypothesis
hypothesis
hypothesis
s hypothesi
hypothesi s
hypothesis
s
hypothesi
hypothesis hypothesi
s
hypot
esis hesis hypothesis
h y po th
B) Write down problem hypothesis (1)
“I believe [type of person]

experience [type of problem]
while doing [type of task]”
Or

because of [limit or constraint]”
Example
“I believe [young music listeners]

experience [inability to conveniently and instantly
access the music they want]
because [existing solutions are too complicated and
limited]”
Example
“I believe [tech operations teams]

experience [wasted time and budget]
while [predicting and securing IT resources for their
companies]”
Problem Hypotheses

Or Specific >> General

because of [limit or constraint]”
Write down problem hypothesis (2)
For (target customer)

Who (statement of need or opportunity),
(Product name) is a (product category)
That (statement of key benefit).
Unlike (competing alternative)
Our Product (statement of primary differentiation).
Example
Example
For World Wide Web users

Example

Who enjoy books,
Example

Who enjoy books,
Amazon is a retail bookseller
Example

Who enjoy books,
That provides instant access to over 1.1 million books.
Example

Who enjoy books,
Unlike traditional book retailers,
Example

Who enjoy books,
Unlike traditional book retailers,
Amazon provides a combination of extraordinary
convenience, low prices and comprehensive
selection.
Example
Example
The only motorcycle manufacturer

Example

That makes big, loud motorcycles
Example

For macho guys (and “macho wannabes”)
Mostly in the United States
Example

Who want to join a gang of cowboys
In an era of decreasing personal freedom.
Example

Who want to join a gang of cowboys
In an era of decreasing personal freedom.
(Official internal positioning statement)

Specific >> General
For (target customer)
Our Product (statement of primary differentiation).
C) Create a Customer Persona
Persona Template
Persona type
Name
Age
Location
Technical comfort
Job Title
Feel free to doodle!
Back story
Tell us a bit about their lives
•
•
•
•
Motivations
What concerns do they have? Why do they need this website/service? How have they found or heard about the website?
•
•
•
•
Frustrations
What’s stopping them from choosing the service/website or annoying them?
Their ideal experience Quote

Their story including features and content which will help them have a great Sum up their experience with the
experience website/organisation/service.
Positive or negative.
•
• “
•
•
_”
WS - Formulate for your project

Or Specific >> General

”For (target customer)
Our Product (statement of primary differentiation).”
Identify assumptions
A) Surface all Assumptions

1. Individually
2. As a group
B) Write down Problem Hypothesis
C) Create Customer Persona
BM-framework
hypothesis
hypothesis
s
hypothesi
hypothesis
hypothesi
s
s
hypothesi
hypothesis hypothesis hypothesi

s
hypot
hesis
y po th esis hypothesis
h
But,
Realize it’s just Hypotheses!
guess
guess
guess
guess guess
guess
guess
guess guess
guess
guess guess
Persona Template
Persona type Guess

Name Guess
Age Guess
Location Guess
Technical comfort Guess
Job Title Guess
Feel free to doodle!
Back story
Tell us a bit about their lives
•
• Guess
•
•
Guess
Guess
Motivations
Guess
What concerns do they have? Why do they need this website/service? How have they found or heard about the website?
•
•
• Guess
•
Guess
Frustrations
Guess
What’s stopping them from choosing the service/website or annoying them?
• Guess
• Guess
• Guess
•
Guess
•
Guess
Their ideal experience Quote

Their story including features and content which will help them have a great Sum up their experience with the
experience website/organisation/service.
Positive or negative.
• Guess
• Guess “ Guess
• Guess Guess
• Guess Guess
• Guess Guess
Guess Guess
•
•
Guess Guess _”
Guess
I believe [guess]
experience [guess]
while [guess]
For (guess)
Who (guess),
(Product name) is a (guess)
That (guess).
Unlike (guess)
Our Product (guess).
Test Problem Hypothesis
guess
guess
guess
guess guess
guess
guess
guess guess
guess
guess guess
If customers could talk…
If customers could talk…
Here’s why it’s a problem!
Here’s how we try to deal with it now!
Here’s who is involved!
Here’s what we’re willing to pay to solve it!

Suggested Flow
1) Person
2) Problem
3) Solution
1. Person
Learn about them and their role in your industry
• What is your name and role at your

company?
• How do you fit into your company’s
structure?
• What is your budget like?
• Who has to approve your purchases?
• How do you discover new products for work?
• Do you need any approval to try them?
• What is a typical day like on your job?
• How much time do you spend on [relevant
task that they mentioned]?
2. Problem
Do they have the problems you think they have?
• What are your top 3 problems (related to X)?
• How do you deal with them today?
• What do you like about your current
solution?
• What did you originally hope to do with it?
• What other solution have you tried in the
past, that were better or worse?
• Who is involved?
• How long does it take?
• Can you describe a typical day when the
problem occurs?
• If you could wave a magic wand…what
would the solution be?
2. Problem
“MTV told me they needed analytics and reports
for their campaigns.”
“I made a big mistake by accepting the feature

request at face value.”
r b l
“Oh yeah, ofocourse.e m I mean, nobody even reads
h e p
T Our clientsejust o n e )
these.
o- b - d like to get something emailed
to them
( jo b-
at t
the end of every
io n a l!
week and we think
fu n c t
was not
they’d be happier if it was a bit fancier, you know?”
“90% of what we had built was irrelevant”
The job-to-be-done was not a functional…

2. Problem
• What are top 3 problems (related to X)?
• How do you deal with them today?
g yo u r
• What do you like about
he ar i n
your current
you
l e
solution/process?
i s ti l l a ti on
m w h e va l i d
ad th e
What did you e m or
originally hope to do with it?
yo u le •
ne d t h
e le s s m e n t
Is i o
there some other solution/process you’ve
Th be i n g •
o b l e m tried in the past that was better or worse?
pr • Who is involved with this solution/process?
hav e !
How long does it take?
• Can you describe a typical day when the
problem occurs?
• If you could wave a magic wand…what
would the solution be?
3. Solution
See if your idea is attractive & get feedback
• Walk them through you product concept
w o r k.
& the problems you think
ati on s you are
o
solving.cDon ver
they s
agree?
le m s,
d pro b
• Would you be willing to pay y for
. such a
efine e ea rl
For w ell d pro t otyXpbe reasonable?”
solution? “Would
m ,“Would you be willing to start right
–blIfeyes:
ive pro
novat away?”
r an in
Fo • “That’s interesting” = Kiss of Death.
• Read body language, voice inflection and
energy level for signals of interest.
(More on testing solution next time)

3. Solution
See if your idea is attractive & get feedback
• Walk them through you product concept
& the problems you think you are
solving. Do they agree?
u ti on !
• Would you u r s
beol
willing to pay
o b l
for em
such a
ti o n y o th e p r
t m e n solution? “Would
rs to o d X be reasonable?”
Do n o g u n d e
e h avi
• n If yes: “Would you be willing to start
Be for right away?”
• “That’s interesting” = Kiss of Death.
• Read body language, voice inflection and
energy level for signals of interest.
(More on testing solution next time)

Interview tips and examples
Be prepared to go off script
If users get worked up – don’t stay on script,

but follow their lead and drill down!
Emotion = Importance!
Don’t talk
You should be talking as little as possible!
80/20 is a good ratio to aim for.
Don’t fear ”uncomfortable silence”

(let them break it).
Ask brief follow-ups
(remember, they should be takning)
That sounds expensive/inefficient/painful…

Tell me more!
When was the last time that happened?
Can you give an example?
What do you mean by that?
Can you explain that a little more?
What else do you do when that happens?
How do you feel about that?
Parrot back and misrepresent => More elaborate answer

Don’t assume things
Bad:
You love walking your dog!
Good:
How do you feel about walking your dog? (speculation)
Better:
What was it like the last time you walked your dog? (recollection)
Bad:
Good:
Better:
Bad:
Good:
Better:
Don’t avoid the scary stuff!
My startup will fail if X is true.
Ask about X !
Get psyched to hear things you
don’t want to hear
Get people to say things you
don’t want to hear
Give permission to disagree
”Other people say…”
”I was at (company X) and they told me the opposite…”

Casual conversations
”A colleague told me about an idea…”
”I read somewhere…”
=> People do not have to protect your ego.

Casual conversations
”A colleague told me about an idea…”
e d to)
”I read somewhere…” prepar
u l d (b e fo r
o u sh o ec i al l y
This y e , e sp
the t i m .
d o al l d uc ts
e r p r o
=> People
cons u mdo not have to protect your ego.
Always end by asking for a committment
Time
Another meeting to walk through mockup/beta
Committment to use a trial version
Reputation
Set up meeting with team
Introduction to boss, spouse…
Cash
Pre-order
Deposit
LoI
Write up results a.s.a.p.
Take notes!
Write upp results and

extract key insights
immediately after
the interview!
Surprises, Trends,
(In)validated hypotheses
Useful people to talk to
“nice to
1. Have a problem
have”
2. Understand they have a problem
3. Actively searching for a solution

4. Cobbled together an interim solution
5. Committed funds for a solution Jackpot!
Useful people to talk to
1. Have a problem
2. Understand they have a problem
3. Actively searching for a solution

4. Cobbled together an interim solution
5. Committed funds for a solution

23.11.07 - CNB - Experimenting - Identifying Assumptions, Testing Customer-Problem Hypotheses, Interviewing

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

23.11.07 - CNB - Experimenting - Identifying Assumptions, Testing Customer-Problem Hypotheses, Interviewing

Uploaded by

Copyright:

Available Formats

TEK010/TEK656

Learning from Customers

Learning from Customers

(What We Used to Believe

Business Model Operating Plan +

Concept/ Product Dev. Alpha/Beta Launch

Customer Problem: known

Product Features: known iPhone 13 iPhone 14

Concept/ Product Dev. Alpha/Beta Launch

Concept/ Product Dev. Alpha/Beta Launch

- Create MarCom - Hire PR Agency - Create Demand

Concept/ Product Dev. Alpha/Beta Launch

- Create MarCom - Hire PR Agency - Create Demand

- Hire Sales VP - Build Sales

Concept/ Product Dev. Alpha/Beta Launch

- Create MarCom - Hire PR Agency - Create Demand

- Hire Sales VP - Build Sales

Business - Hire 1st Bus Dev - Do deals for Launch

Patrik Hedelin, Executive Chairman.

• Both Customer Problems and Product Features

You do not know if you are wrong until you

- Create MarCom - Hire PR Agency - Create Demand

- Hire Sales VP - Build Sales

Business - Hire First Bus Dev - Do deals for FCS

• We have (and use) processes to manage

• We have no established process to manage

Concept Product Dev. Alpha/Beta Launch

Customer Customer Customer Company

Strategy Business Model

Process Customer & Product Management

No sales, marketing, and business

No sales, marketing and business

Customer Development, Product Management

Customer Functional Organization

More startups fail from

• Both Customer Problems and Product Features

You do not know if you are wrong until you

“We have been too visionary. We

“Experimentation describes how a

Concept Product Dev. Alpha/Beta Launch

Customer Customer Customer Company

Problem: unknown Solution: unknown

Problem: unknown Solution: unknown

Problem: unknown Solution: unknown

• Parallel process to Product Development (agile)

• Progress assessed not in terms of customer insights

• Emphasis on iterative learning before execution

• Must be done by small team including CEO/project leader

• There are no facts inside, so get out of the building!

• Develop for the few, not for the many

• “Earlyvangelists” make your company and are smarter than you!

• Develop a minimum viable product to maximize fast learning.

A) Surface all Assumptions

• Brainstorm/identify hypotheses individually

• Brainstorm/identify hypotheses individually

“I believe [type of person]

“I believe [type of person]

“I believe [young music listeners]

“I believe [tech operations teams]

“I believe [type of person]

Or Specific >> General

For (target customer)

For World Wide Web users