Book Summary: The Lean Product Playbook

Trev de Vroome

Oct 3, 2020·25 min read

‘The Lean Product Playbook’ — by Dan Olsen
The Lean Product Playbook

by Dan Olsen

Resonating Message

To create a successful product — you need to satisfy all layers of the Product-Market fit pyramid,
ensuring you’re clear on who you are targeting the product for, what their needs are, and then
creating a compelling offering that satisfies those needs with a compelling value proposition, feature
set, and UX. The Lean Product Process sets out a six-step process to achieve this:t

1. Determine your target customers

2. Identify underserved or unmet needs
3. Define a compelling value proposition
4. Identifying an MVP feature set
5. Creating your MVP prototype
6. Testing your MVP with customers

Chapter in one sentence

1. Achieving Product-Market Fit with the Lean Product Process: Building a

product isn’t enough. You must ensure you understand the market forces —
and build a product that connects with that market through a product-market
fit.

2. Problem Space versus Solution Space: You must separate and use different
techniques for the problem and solution space — leading first with the
problem spaced focused on emphasizing with and understanding the needs
of your target customer, before delving into the solution space of addressing
those needs.

3. Determine your Target Customer (Step 1): Once you understand the problem
space, your goal is to identify a target customer and associated segment that
you believe has unmet or underserved needs aligned with your target
product vision.

4. Identify Underserved Customer Needs (Step 2) The goal now is to identify and
elaborate on the needs of your customer — identifying underserved or
unmet needs through empirical measurement against the desired outcomes
of the target customer.

5. Define Your Value Proposition (Step 3) You will define your value proposition
by establishing your hypothesized product benefits (must-haves,
performance, and delighters) in comparison to your competitors — both in
the current market and the predicted future to ensure you’re addressing the
most important anticipated needs.

6. Specify Your Minimum Viable Product (MVP) Feature Set (Step 4) Now you
need to shift from the problem space and into the solution space — exploring
how you will serve those underserved customer needs. You will do that with
 divergent thinking to establish user stories, prioritize them on ROI, and shape
a viable MVP to create “something that customers find superior to other
products, and ideally, unique”

7. Create Your MVP Prototype (Step 5) When creating your MVP (minimum
viable product), you need to focus on building a slice of value to the customer
that allows you to test your hypothesis. You must both test the Product &
Marketing, using qualitative and quantitative methods to assess your current
product-market fit

8. Apply the Principles of Great UX Design We must carefully consider the UX in

how it helps or hinders the customers in realising the value proposition of the
product. This is done by establishing who the Product is for and what they
need, designing interactions that allow them to obtain that experience with a
visual design that makes consumption simple through standards, principles,
consistency, and simplicity.

9. Test Your MVP with Customer (Step 6) Once you have an MVP, you need to
test it with your customer — getting qualitative feedback on how your
functionality, UX, and messaging come together to address the Product-
Market Fit. Recruit a user in your target market, guide them through user
testing, and collate your results to inform the next steps for your Product.

10. Iterate and Pivot to Improve Product-Market Fit Test your hypotheses in
iterations through the hypothesize-design-test-learn loop —incorporating
your learnings into refining your hypothesis for the next iteration. As you do,
measure your progress towards Product-Market fit — carefully consider if it’s
time to pivot if your progress towards fit slows.

11. An End-to-End Lean Product Case Study

<This chapter demonstrates the learnings from the prior chapters>

12. Build Your Product Using Agile Development Just as we tested our
hypotheses in fast learning loops, must we also develop our Product in
iterative and incremental loops — to incorporate the new validated learning
we’ll receive as customers user our newly created product.

13. Measure Your Key Metrics You need to carefully measure your product —
selecting the appropriate metric for the specific insight you’re seeking. Start
with measuring Product-market fit via retention rate, then build a model of
measurement relevant to your Product to monitor the success and impact of
your Product improvements you make over time.

14. Use Analytics to Optimize Your Product and Business For every live product —
you will constantly battle tension between your product aspirations, and the
limitations your resourcing imposes. Where you spend those limited
resources becomes crucial to your ability to maintain an advantage over your
competition. To do this — ruthlessly prioritize and focus on the most
important metric, rapidly taking that focus through the build-measure-learn
loop in fast iterations to obtain validated feedback on your improvements.
1. Achieving Product-Market Fit with the Lean Product Process (P3–12)

• The goal of the Lean Product process is to leverage the product-market fit
pyramid — working from the bottom up to unpack the market, identify
product-market fit, and then refine the product to validate and test with
customers.

© Dan Olsen 2015

• Identify the market, its size, and target segments to hone in on identifying
your target customer — then focus on unpacking their specific needs

• The goal is to identify “underserved needs” that you can address through your
proposed product.

• The five layers of the Product-Market Fit Pyramid are:

1. Target Customer - The specific segment and customer on which you're targeting to deliver value
2. Underserved Needs - Areas of the target customers needs that aren't adequately met
3. Value Proposition - An innovation, service, or feature intended to make the product attractive to your
target customers
4. Feature Set - The set of features your MVP product provides
5. UX - What the customer sees and interacts with to consume your product or service

• “product-market fit … is the measure of how well your product (the top three
layers of the pyramid) satisfies the market (the bottom two layers of the
pyramid)”
 • The Lean Product Process
1. Determine your target customers
2. Identify underserved customer needs
3. Define your value proposition
4. Specify your minimum viable product (MVP) feature set
5. Create your MVP prototype
6. Test your MVP with customers

• You must intelligently apply the principles and concepts of the Lean Product
Playbook to the challenges unique to your business

• Chapter Principles:
1. Establish an undertanding of the market and underserved needs within it
2. Develop and test a product to address those underserved needs
3. Product-Market fit will come when the product or service you provide satisfies the market
4. The lean Product Process iterates to create and test your hypothesis on how to solve product-market fit
with your product

2. Problem Space versus Solution Space (P13–24)

• The Problem Space is the understanding of the needs, pains, gains, and jobs-
to-be-done of your target customer.

• The Solution Space is the products or services you create to address those
target customer needs.

• Separate the “why and what” (problem space) from the “how” (solution
space)

• “By talking with customers to understand their needs, as well as what they
like and don’t like about existing solutions, outside-in product teams can form
a robust problem-space definition before starting product design”

• You need to apply product thinking & design to turn customer needs into
products and services with product-market

• Customers will often tell you their concerns by providing solution space
answers — it’s key you facilitate unpacking the needs behind those, to deeply
understand the solution space — and provide space to truly innovate to the
root cause needs.

• Chapter Principles:
1. Keep the problem space free from solutions
2. A clear problem space will more clearly define its target market
3. Ensure you talk to customers to understand their needs
4. Apply product thinking to innovatively address customer needs
5. Use the solution space to discover the problem space

3. Determine your Target Customer (Step 1) (P25–36)

 • “Define your target customer by capturing all of the relevant customer
attributes that identify someone as being in your target market”

• You may find your product only needs minor changes to also reach product-
market fit for adjacent markets

• You can segment a market based on:

1. Demographics
2. Psychology
3. Behaviour
4. Needs

• Where the person using the product and purchasing the product — you
should define your target buyer in addition to your target customer, and seek
to find product-market fit for both.

• Be mindful of the technology adoption lifecycle — the segments that

embrace innovation and early adoption will have different needs than those
to break through into a market.

• “Good personas convey the relevant demographic, psychographic, behavioral,

and needs-based attributes of your target customer”

• Don’t make your persona an “average” of your typical target customers.

Instead, base it off a real customer that well represents someone close to
your average — and give them a name to ground the persona in reality.

• Chapter Principles:
1. Capture relevant attributes for your target customer
2. Identify your target market segment
3. Consider the needs of both the target user and target buyer
4. Understand where your target customer is in the technology adoption lifecycle
5. Use personas to establish common understanding of your target customer(s)

4. Identify Underserved Customer Needs (Step 2) (P37–66)

• “Customers are generally not skilled at discussing the problem space; they are
better at telling you what they like and dislike about a particular solution.”

• Customer needs are layered, nuanced, and rarely directly articulated as

problems and not solutions — the goal of a good product professional is to
tease out to find the root cause of the customers' problem space.

• “You’ll also notice that each benefit begins with a verb: help, check, reduce,
maximize. A benefit conveys value, which means it’s doing something for the
customer.”

• In interviews — ask sets of open questions about each hypothesised benefit

statement to tease out the root cause
 • Be aware of the hierarchy of needs of your users — you won’t be able to
address higher-level needs until their base needs are met.

© Dan Olsen 2015

• Plot needs against two axes — importance and satisfaction, looking for areas
of low satisfaction and high importance to inform opportunities.
© Dan Olsen 2015

• Disruptive innovation sees satisfaction shift left of existing products, being

displaced by a new product setting a new benchmark for satisfaction.

• Obtain data from your target customers to quantify importance and

satisfaction

• Scales of 5 or 7 responses for bipolar responses are ideal for measures where
there can be positive, neutral, and negative responses. Unipolar scales (ie. 1
to 10) are better used to determine a measure which is a degree.

• Form a hypothesis on your target customers needs, then inform and update it
with empirical data over time.

• Some key calculations to identify and assess outcomes:

1. Opportunity Score = Importance + Maximum (Importance − Satisfaction, 0)
2. Customer Value Delivered = Importance X Satisfaction
3. Opportunity to Add Value = Importance X (1 − Satisfaction)
4. Opportunity = Importance − Current Value Delivered
5. Customer Value Created = Importance X (Satisfaction(after) − Satisfaction(before))

• The kano model splits needs into three categories:

1. Performance - More is always better
2. Must haves - Creates dissatisfaction by not being met, as it is a fundamental expectation
3. Delighters - Unexpected benefits that exceed expectations (but only when must haves are first met)

© Dan Olsen 2015

• Chapter Principles:
1. Work through the customers' problem space to identify their needs
2. Use discovery interviews to test your benefit hypotheses
3. Leverage benefit ladders to find the root need using the 5 whys
4. Address needs at the lowest point in the customer hierarcy first
5. Find opportunity in important needs with low satisfaction through measurement and validation
6. Establish a hypothesis for your target customer upon which you build and update with emperical data
7. Quantify the size of opportunities (eg. using Jobs-to-be-done)
8. Focus on identifying outcomes, not needs or benefits through emperical measurement
9. Use the kano model to identify your must have, performance, and delighter needs

5. Define Your Value Proposition (Step 3) (P67–76)

• Focus on a small set of the most important needs to shape your MVP

• Everything until this point, however quantitative, is based on an assumption

of your target customer. Keep your MVP hypothesis small, rapidly test, and
adjust as necessary.
You will never have “perfect information.” If you are following a good trajectory as you iterate, there
will just be “less imperfect” information that you gather with increasing confidence.

• To compete with an incumbent market leader in a mature space, you need to

ensure you match both the must-haves and performance standard — but you
must then differentiate yourself beyond that benchmark, creating excess
value to compel them to switch.

• The value proposition template works as follows:

- List all benefits (must-haves, performance, and delighters) relevant to you and your competitors
- List all competitors (direct and indirect)
- The the benefits of each of your competitors, and your hypothesized product
- Highlight the differentiators (performance & delighters) of each competitor and your hypoethesized
product
- Prodict the future states of each competitor and your hypoethesized product
- Use the differentiators to articulate the value proposition of your product vs its competitors
- Analy

• “To be strategic, you want to ensure that you are projecting forward in time,
anticipating the important trends in your market and what competitors are
likely to do”

• Chapter Principles:
1. Identify your value proposition
2. Decide on what you're not going to do
3. Focus on what matters to your target customer
4. See how your value proposition stacks up against your competitors
5. Ensure your value proposition projects forward in time
6. Analyze your product strategy against the currrent and future market

6. Specify Your Minimum Viable Product (MVP) Feature Set (Step 4) (P77–88)

• “For your MVP, you want to identify the minimum functionality required to
validate that you are heading in the right direction”

• Start by breaking down your product into use stories to capture all aspects of
the benefit using who (as a), what (I want), and why (so that)

• Use story points to estimate — identifying and large stories, aiming to

continually break them down into small bite-sized deliverables using the
INVEST acronym

• It’s not enough to prioritize on value alone — you need to weight value vs
cost, to surface a true return on investment (ROI) from which to weight your
decisions

The goal is to to create high customer value for low overall effort
 • Both the estimates of value and estimate of cost (story points) are guesses at
this point — instead, the critical point is how they compare

• Your MVP should have just enough of each of the three kano categories to
provide “something that customers find superior to other products, and
ideally, unique” — including:
1. All the must-haves
2. The main performance benefit you're planning to use to beat the competition
3. Your top delighter (if your performance benefit isn't a strong enough differentiator)

• Chapter Principles:
1. Use User stories to capture and prioritize customer benefits
2. Use chunking to break down features
3. Keep batch sizes small
4. Use story points to estimate; breaking down large stories into smaller, more incremental pieces
5. Prioritize based on return on investment
6. Establish the minimum set of functionality reuqired for your MVP candidate
7. Don't plan too far ahead - plans will need to change once you get real customer feedback

7. Create Your MVP Prototype (Step 5) (P89–110)

• How much you build (landing page vs real product) is entirely dependent on
the hypothesis you wish to learn more about — and will be different for each
product.

• In this chapter — MVP is described as the MMF (minimum marketing features

— the least amount of working functionality that a customer would need to
realize any value) — as opposed to the alternate definition of MVP being
“MVP Tests”, that aim only to buy information on the hypothesis

• There are two key types of tests:

1. Product Tests - Validating the product through a live beta or wireframes to assess product/market fit
2. Marketing Tests - Validate the prospective functionality through marketing sites and conversion data to
assess needs

• And two dimensions of testing we can use:

1. Quantitative - Testing at scale using objective numbers to quantify the 'What' and 'How many' of a given
hypothesis
2. Qualitative - Testing 1:1 to understand and empathise with a customer to get the 'Why' behind that
customers sentiment

• When starting a new product “it is most beneficial to start with qualitative
tests to gain some initial understanding”

• Quantitative Marketing Test Examples:

1. Landing Page/Smoke Test
2. Explainer Video
3. Ad Campaign
4. Marketing A/B Testing
5. Crowdfunding

• Qualitative Product Test Examples

1. Hand Sketches
2. Wireframes
3. Mockups
4. Interactive Prototype
5. Wizard of Oz
6. Concierge
7. Live Product

• Quantitative Product Test Examples

1. Fake Door/404 Page
2. Product Analytics
3. A/B Tests

• Chapter Principles:
1. Build an MVP prototype that delivers value to your customer
2. Validate the hypothesis with MVP tests
3. Use a blend of product/marketing tests
4. Use quantitative tests to understand 'what' and 'how many'
5. Use qualitative tests to understand 'why'

8. Apply the Principles of Great UX Design (P111–142)

• “User experience (UX)… brings your product’s features and benefits to life for
the customer. Even if you have made good decisions on the other four layers,
you will not achieve product-market fit without a good UX.”

• “Poor UX gets in the way, preventing the user from realizing the benefits.
Great UX makes it easy for the user to realize the benefits that the product’s
functionality offers.”
Usability — Can your customer use the product
Delight - Do your customers enjoy using your product?

• Use divergent thinking for ideation, and convergent thinking for evaluation
and selection from those ideas

• Olsen’s Law of Usability:

The more user effort required to take an action, the lower the percentage of users who will take that
action.The less user effort required, the higher the percentage of users who will take that action.

• Some common methods of creating delight:

1. Simpilcity
2. Smart defaults & automation
3. Evoking emotion (eg. humour)
4. Dynamic response (eg. sound & animation)
5. Suprise

• The UX Design Iceberg sits on top of the Product-Market Fit Pyramid in place
of the UX layer:

© Dan Olsen 2015

1. Conceptual Design - Identifying a target customer and designing a concept that will describe how the
feature set will support how they think and act
2. Information Architecture - Define how information and functionality with be grouped and navigated
including navigation and sitemaps
3. Interaction Design - Define how the product and user will interact with one another in terms of
inputs/outputs, operating modes, and processes
4. Visual Design - Define the color palatte, typography, and graphics to make the interactions come to life.

• Consistency is key for great UX — creating a predictable layout, naming

conventions, inputs and interactions, colours, typography, and graphics
throughout your Product.

• Use Style Guides and Layout Grids to drive consistent design across variations
internally (different engineering teams) or externally (different browsers or
devices accessing the experience)

• In many cases, there are many screen sizes you may need to accommodate in
your product. If mobile is key to your customer — consider what the mobile
 experience will be, develop a responsive design, and lead the design by
mobile in parallel with the desktop experience.

Great UX needs a team skilled in product management (who the product is for, and what they
need), visual design (how the product looks), interaction design (how the product works), and front-
end development (how the design will be implemented)

• Chapter Principles:
1. Be mindful of "Olsen's Law of Usability"
2. Look for ways to create Delight
3. Address all layers of the UX Design Iceberg
4. Focus on consistency throughout your UX
5. Be mindful of common design principles (Gestalt, heirarchy, composition, responsiven design etc.)
6. Copy is part of UX, and also needs careful consideration

9. Test Your MVP with Customer (Step 6) (P143–165)

• Being close to the product means you can develop “product blindness” —
where you don’t see and experience the product as a user does, overlooking
the pains/gains experienced by a new user.

• Groups can bias your testing feedback — if you must test in a group, use tools
like Liberating Structures⁵ to give everyone an equal and unbiased voice.

• There are two key qualitative testing approaches:

1. Moderated - Research is present and conducting the test with the customer
2. Unmoderated - No moderate is present, but customers are given instructions on what to do.

• And there are two key qualitative testing mediums:

1. In-person - Being with the customer in person
2. Remote - Remotely testing via video call/screen sharing etc.

• “When you are early in defining and validating your MVP, moderated testing
is the way to go to ensure you can ask questions and get rich customer
feedback.”

• “When you are farther down the road and feeling more confident about your
MVP”

• Use Screener questions, linked to personas to quickly identify those users

aligned with your target personas for whom you’re seeking feedback.

• Some methods of feedback recruitment:

1. Online crowdsourcing (Amazon MTurk, Upwork etc.)
2. Kickstarter
3. Conferences
4. Coffee Shop/Mall
5. Follow-me home

• To kick start your testing, use this basic user testing approach:
1. Find dedicated space you already have
2. Use someone from the team to moderate the feedback
3. Use a dedicated note-taker
4. Have the customer bring their own preferred device
5. Sit next to them
6. Have a script prepared and practiced and tested beforehand
7. Keep the test to 1 hour +/- 15 minutes
8. Have an intro & warm-up (up to 15m), feedback (up to 45m), then wrap-up and closing questions (up to
15m)

• “it’s important to explicitly tell users upfront that you want their honest
feedback, even if it’s negative”

• Encourage the user to share their thoughts out loud — do this by asking
good, open questions.

• It’s encouraged to use two questions to wrap up your test:

1. “Would you be willing to participate in future research?”
2. “Would you like to be notified when this product is available?”

• Capture your feedback in a structured manner, categorizing under:

1. Functionality - Does your MVP address the right benefits?
2. UX - Does your MVP let users take advantage of your features?
3. Messaging - Dooes the way you talk about your MVP reasonate with custmoers?

It’s crucial as you conduct your user tests to differentiate between feedback on usability versus
product-market fit.

Feedback on usability has to do with how easy it is for customers to understand and use your product,
whereas feedback on product-market fit has to do with how valuable they find your product.

• UX and messaging can prevent the user from experiencing the functionality
— but only the functionality can determine if there truly is product-market fit.

• Chapter Principles:
1. Conduct the test with one customer at a time
2. Test in waves of 5-8 customers to keep feedback cycles small
3. Target your recruitment of testers to your target personas
4. Anyone can do the testing, if it's structured properly
5. Be well prepared, engaging, and take ~1h for each user test
6. Avoid intervening and bias by using open questions
7. Ensure the customers you're talking to are in your target market
10. Iterate and Pivot to Improve Product-Market Fit (P167–179)

• The goal of testing and assessing your MVP for product-market fit involves
rapidly iterating through the build-measure-learn loop⁶ to obtain validated
learning on your hypotheses.

“Build” simply means having something that you can test with customers, which could be a live
product or design artifacts, such as wireframes or mockups

• But the build-measure-learn loop⁶ fails to clarify key points:

- You don't need to build something, you just need something to test
- Measure can be both qualitative and quantitative
- Learn needs to not only be capturing the learning, but to modify your hypotheses based on that learning

© Dan Olsen 2015

you test and improve your problem space thinking by showing customers a product or design artifact
in thesolution space and soliciting their feedback on it.

• The lower down the Product-Market pyramid you need to adjust to

incorporate your validated learning — the most costly. Focus on lower layer
feedback first.
 • It’s important to synthesize your learnings into a structure that allows you to
draw statistically meaningful insight out for each iteration of validated
learning you capture through the loop.

• Find a balance between over-testing the product, and releasing too early and
causing major rework — an MVP should be minimal, but it should also be
viable.

• If you’re not making progress as you iterate:

1. Pause and step-back
2. Map each problem to the layer of the Product-Market Pyramid
3. Re-focus on the lowest layer problem you have
4. Change your incorrect hypotheses

• A change to a main hypothesis is called a pivot

A pivot is larger in magnitude than the change you normally see as you iterate along the path you
have chosen; it means a significant change in direction.

• If you aren’t making the necessary gains towards Product-Market fit — you
should consider a Pivot. Their types⁶ include:
1. Zoom-in - Focus deeply on one key feature
2. Zoom-out - Focus widely on more features
3. Customer Segment - Change the Customer Segment you target
4. Custoner Need - Focus on a different need or job for the customer
5. Platform - Switch from an App to a Platform
6. Business Architecture - High-margin/low-volume to low-margin/high-volume
7. Venture Capital - Switch how you monitize or earn revenue
8. Engine of Growth - Switch growth model (virual/sticky/paid)
9. Channel - Change how and where you sell your product
10. Technology - A new technical to achieve the same outcome

• Measure the rate of progress you make towards Product-Market fit each time
you iterate. If you can’t find other paths to Product-Market fit for your
current hypotheses — it may be time to pivot to another set of hypoetheses.

• Chapter Principles:
1. Use the hypothesize-design-test-learn loop
2. Address issues lower in the Product-Market pyramid first
3. Conduct iterative user testing
4. Sythesize user testing results to create insights
5. When necessary, pivot

11. An End-to-End Lean Product Case Study (P181–198)

<This chapter demonstrates the learnings from the prior chapters>

12. Build Your Product Using Agile Development (P201–228)

• Just as we tested our hypotheses in fast learning loops, must we also develop
our Product in iterative and incremental loops — to incorporate the new
validated learning we’ll receive as customers user our newly created product.

• User the User Story format to capture requirements in a customer outcome-

focused way:
As a [type of user],
I want to [do something],
so that I can [desired benefit].

• Scrum is a method that focuses on delivering in time-boxed increments called

a Sprint. Its characteristics include:
- Value delivered in iterations of up to 4 weeks
- Teams between 5-9 people
- Three key roles: Product Owner, Scrum Master, and Development Team
- The Product Owner defines and prioritises the work
- The Scrum Master coaches and facilitates the process
- The Development Team develop the prioritised increment
- Ceremonies include: Daily Scrum, Sprint Planning, Sprint Review, and the Sprint Retrospective
- Needs are captured as Stories & estimated using Fibonacci numbers
- Key metrics are value, velocity, and sprint success
- Delivery monitored by a burn-down
- Each iteration delivers a potentially shippable product
- The team inspects and adapts after each iteration

• Kanban is a method that focuses on continuous flow and visualization of the

work. Its characteristics include:
- Value delivered continuously in a flow across a series of steps
- Highly visual 'Kanban' board visualizes those steps
- The work in the system is limited via a work-in-progress limit
- Work is pulled across the stages when capacity is free to process it at the next stage
- Key metrics are throughput, cycle time, and lead time

• To succeed, you want to have a cross-functional team working on a clear and

prioritized backlog, regularly delivering small increments of value to build and
test your Product Hypothesis.

The trick is to be both rigid and flexible when it comes to prioritizing your backlog. You must be clear
on your rank order priorities at any point in time; but you must also be able to quickly incorporate new
or changing requirements.

• The quicker you find and resolve defects, the fewer negative impacts they’ll
cause to customers and to your teams rework and handling effort.

• Quality assurance methods include:

- Code reivew - Reviewing a peers code for mistake
- Pair programming - Two developers working together to code & review at the same time
- Validation Testing - Checking new functionality to see if it works as expected
- Regression Testing - Confirming that existing features continue to work as new changes are incorporated
- Test-driven Development - Creating failing tests, writing code to pass the test, then refactoring that code
for cleanliness

• Use Continuous Integration to continuously test and validate your source

code when changes are made — to rapidly assess the state of the code.

• Use Continuous Deployment to automatically deploy code that passes all

tests to an environment for use and validation — carefully tracking metrics
and telemetry to detect and automate rollback (if necessary)

• Chapter Principles:
1. Use Agile Delivery Practices
2. Keep batch sizes small
3. Use the Agile Delivery method that fits your organisation
4. Train your team in Agile
5. Establish cross-functional collaboration
6. Ruthlessly prioritise
7. Use the User Story format
8. Keep stories small and deliver value continuously
9. Empoloy quality assurance techniques
10. Use Continuous Integration/Continuous Deployment

13. Measure Your Key Metrics (P229–258)

• Before launch, you have to rely on more qualitative data — however post-
launch, you use quantitative tools to start capturing indirect behaviour.

• It’s important to balance the right research methods and metrics to obtain
validated learning and test your hypothesis — using methods across the
Research Methods Framework:
© Dan Olsen 2015

• Attitudinal information is what customers say,

whereas behavioural information is what customers do.

Quantitative research can tell you how many customers are doing (or not doing) something. But it
won’t tell you why the customers are doing it (or not doing it).

• Surveys are sensitive to the wording of the question and the knowledge of
the respondent — use them with careful consideration.

• Some recommended use cases for specific research methods include:

User Interviews - Customer needs and preferences
Usability Testing - Customer behaviours within a Product
Surveys - Customer attitudes over time
Net Promotor Score (NPS) - Customer satisfaction with your Product

you should not invest in trying to grow your business until after you have achieved product-market fit

• You can use Sean Ellis’ Product-Market Fit question to assess your level of
fit: “How would you feel if you could no longer use [product X]?

• A/B testing what tell you why a customer behaviour exists, but it will provide
objective and comparative data on what the customer behaviour is for a
given change

Quant was the smoking gun that told us we had a conversion problem and identified where people
were dropping off the most, but it couldn’t tell us why.

Qual gave us the insights we needed to understand and address the issues. After we rolled out the
improvements, quant showed us the impact of the changes we made.

• Use the AARRR framework gives you conversion funnel metrics for your
Product — tracking 2–3 metrics per layer:
© Dan Olsen 2015

• At any given time — there will be one clear metric that matters most (MITM)
across the conversion funnel — it will be the metric that provides the largest
ROI through the funnel.

• You need to optimise your product in this order:

- First optimise for retention, until you find Product-market fit with those first few Customers
- Then optimise for converstion, ensuring Customers who find your Product go on to adopt it
- Once optimised for retention and conversion, focus then on acquisition to find new Customers

Retention rate is the single best metric to measure your product-market fit.

• Calculate the retention rate by dividing the number of active customers by

the total number of customers — monitoring across three key metrics:
- Initial drop off rate - How many users drop-off on day 1
- Rate of descent - The rate at which retention decreases down to a terminal value
- Terminal value - The value at which the retention flattens out over time

• Cohort analysis measures a group of users that share a common

characteristic over time — it is used best to measure the changes to Product-
market fit over time by looking at the differences in retention of different
groups of users.

• You need to understand the appropriate metrics for your specific business —
expressing this as an equation that can be monitored and optimized from a
set of relevant metrics.

• Once you have identified your specific business equation — break it down
(aka. “peel the onion”) by to its component parts to identify the specific
metrics that drive the equation. This will enable you to monitor exactly where
you need to improve your product — driving prioritisation of improvements.

• Chapter Principles:
1. Use the best-fit research method for the insight you're seeking
2. Use quantitative insight to tell you understand where your issues are, and qualitative insight to tell you
what they are
3. Establish an analytics framework to monitor your Product
4. Use the AARRR framework to monitor your conversion funnel
5. Identify and focus on your MITM (Metric that matters most)
6. Optimise for retention rate, then conversion, then retention
7. Identify, measure, and optimise the equation of your business
8. Breakdown your business equation into components to measure

14. Use Analytics to Optimize Your Product and Business (P259–275)

• Once your Product is live — you need to identify and focus your build-
measure-learn loops on addressing the most important metric to maximise
the efficiency of your resources.

The size of your current business becomes less relevant; instead, how quickly you can learn from
customers and iterate becomes the basis of competition.

• Follow the ‘Lean Product Analytics Process’ by performing the following

steps:
© Dan Olsen 2015

• For each metric you try and improve — you should have an informed
hypothesis of how much that is expected to translate to your top-level metric
(ie. revenue), and use those to project and prioritise based on ROI to find
your MTMM (Metric that matters most)

• Using an A/B testing framework will allow you to objectively assess the
impact of your changes compared to your baseline.

• Be wary of ‘atomic’ metrics (ie. the total number of daily users) — instead, try
and use ‘normalized’ ratio metrics where possible to enable objective
comparison across cohorts and time.

• Use the upside potential — the maximum possible improvement that could
be possible — to help inform the ROI of possible improvements by looking for
those metrics with high upside improvement.

• When comparing quantitative results (either realtime A/B testing metrics, or

before/after metrics) — ensure you develop statistical significance in your
results

Quantitative testing will tell you what, but they won’t always tell you why — use qual to understand
the motivation behind the behaviour
 • Chapter Principles:
1. Define your key metrics
2. Measure baseline values for metrics
3. Evaluate ROI potential for each metric
4. Select top metric to improve
5. Rapidly iterate towards improving the metric