Lean

Product
Analytics
Dan Olsen
Olsen Solutions

Feb 5, 2014
 My  Background  
n Educa7on  
n BS,  Electrical  Engineering,  Northwestern  
n MS,  Industrial  Engineering,  Virginia  Tech  
n MBA,  Stanford  
n Web  development  and  UI  design  
n 20  years  of  Product  Management  Experience  
n Managed  submarine  design  for  5  years  
n 5  years  at  Intuit,  led  Quicken  Product  Management  
n Led  Product  Management  at  Friendster  
n CEO  &  Cofounder  of  YourVersion,  “Pandora  for  your  news”  
n Consultant:  Box,  YouSendIt,  Chartboost,  One  Medical  
 
Will  post  slides  at  hUp://slideshare.net/dan_o    

Copyright  ©  2014  Olsen  Solu7ons  

 What  does  “Lean”  mean?  
n Lean  Startup  
n Achieving  product-­‐market  fit  
n Tes7ng  hypotheses  &  learning  
n Valida7ng  MVP  with  users  
n Improving  &  itera7ng  your  
product  quickly  
n Minimizing  waste  =  
using  resources  effec7vely  

Copyright  ©  2014  Olsen  Solu7ons  

 What’s  the  Formula  for  
Product-­‐Market  Fit?  
n A  product  that:  
n Meets  customers’  needs  
n Is  beUer  than  other  alterna7ves  

n Is  easy  to  use  

n Has  a  good  value/price  

Copyright  ©  2014  Olsen  Solu7ons  

 Dan’s  Model  for  the  Causality  
Underlying  Product-­‐Market  Fit  

Customer  
Customer  
has  needs   Needs  

Target   You  design  &  

build  product  
Customer   to  meet  needs  

Customer  
decides  how  
well  product   Product  
meets  needs  
(sa7sfac7on)  
Copyright  ©  2014  Olsen  Solu7ons  
What  are  Customers  Reac7ng  To  
When  They  Use  Your  Product?  

Feature  Set  

UX  Design   Messaging    

Copyright  ©  2014  Olsen  Solu7ons  

Valida7ng  New  vs.  Exis7ng  Products  
New  Product   Exis0ng  Product  
Qualita7ve  interviews   Quan7ta7ve  data  

Oprah   Spock  
+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+

How to be a
Lean Product
Ninja

slideshare.net/dan_o/

+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+
Copyright  ©  2014  Olsen  Solu7ons  
 Lean  Product  Analy7cs  Process  

Iden7fy  What  
Your    Metrics  Are  

Iden7fy  highest  
ROI  idea  
Measure  Metrics  
Baseline  Values  

Evaluate  Metrics   Select   Brainstorm   Learn   Design  and  

Ideas  to  
Upside  Poten7al   Top  Metric   &  Iterate   Implement  
Improve  Metric  

Analyze  How  
the  Metric  
Changes  
Global   Metric  
Level   Level  
Copyright  ©  2014  Olsen  Solu7ons  
Valida7ng  Product-­‐Market  Fit:  Surveys  
n Net  Promoter  Score  

Key  follow-­‐up  ques7ons:  

• Why  did  you  give  the  score  you  did?  
• What  do  we  need  to  do  to  improve?  
Copyright  ©  2014  Olsen  Solu7ons  
Qualita7ve  Compliments  Quan7ta7ve  

Quant
What? Qual
Why?

Copyright  ©  2014  Olsen  Solu7ons  

Valida7ng  Product-­‐Market  Fit:  Surveys  
n Survey.io  /  Qualaroo.com  
n “How  would  you  feel  if  you  could  no  longer  use  Product  X?”  
n Very  disappointed  
n Somewhat  disappointed  
n Not  disappointed  

n General  guideline:    40%  or  more  “very  disappointed”  =  

product-­‐market  fit  

Copyright  ©  2014  Olsen  Solu7ons  

 Product-­‐Market  Fit:  
Actual  User  Behavior  Trumps  Opinions  
n Asking  a  user  ques7ons  in  an  interview  or  survey  
n Valuable,  but…  
n They’re  telling  you  what  they  think  they  would  do  
n Measurement  bias  (because  you’re  with  them)  
n Observing  behavior  
n See  what  users  actually  do  
n Without  you  there  
n Behavioral  metrics  for  Product-­‐Market  Fit:  
n Prospects  sign  up  =  High  conversion  rate  
n They  keep  using  it  =  High  reten7on  rate  
n They  use  it  omen  =  High  frequency  of  use  
n They’re  deeply  engaged  with  it  =  Long  session  7mes  
n They  pay  for  it  =  Revenue  per  customer  

Copyright  ©  2014  Olsen  Solu7ons  

 Valuable  to  Have  a  Holis7c  
Analy7cs  Framework  
Dave  McClure’s  “Startup  Metrics  for  Pirates”  
A
A
R
R
R
Focus  on  right  metric  at  right  7me  
 Using  Analy7cs  for  Op7miza7on  
n In  addi7on  to  Product-­‐Market  Fit,  you  
can  apply  the  Lean  Product  Analy7cs  
Process  to  op7mize:  
n Your  Business  Results  

n Your  User  Experience  

Copyright  ©  2014  Olsen  Solu7ons  

 Define  the  Equa7on  of  your  Business  
“Peeling  the  Onion”  
Adver7sing  Business  Model:  
 Profit  =  Revenue  -­‐  Cost  
 
   Unique  Visitors    x    Ad  Revenue  per  Visitor  
 
         Impressions/Visitor    x    Effec7ve  CPM  /  1000  
 
         Visits/Visitor    x    Pageviews/Visit    x    Impressions/PV  
 
   New  Visitors  +  Returning  Visitors  
 
   Invited  Visitors  +  Uninvited  Visitors  
 
 #    of  Users  Sending  Invites    x    Invites  Sent/User    x    Invite  Conversion  Rate  

Copyright  ©  2014  Olsen  Solu7ons  

 Equa7on  of  your  Business:  
Subscrip7on  Business  Model  
 Profit  =  Revenue  -­‐  Cost  
 
         Paying  Users    x    Revenue  per  Paying  User  
         
 
       New  Paying  Users    +    Repeat  Paying  Users  
           

         Trial  Users    x    C  onv  Rate

       Previous  Paying  Users    x    (  1  –  Cancella7on  Rate  )  
 
         
 
       (    SEO  Visitors  +  SEM  Visitors  +  Viral  Visitors  )    x    Trial  Conversion  Rate  
   

Copyright  ©  2014  Olsen  Solu7ons  

 How  to  Track  Your  Metrics  
n Track  each  metric  as  daily  7me  series  
  Unique   Page   Ad   New  User  
…  
Date   Visitors   views   Revenue   Sign-­‐ups  
4/24/08   10,100   29,600   25   490  
4/25/08   10,500   27,100   24   480  
…  

n Create  ra7os  from  primary  metrics:    X  /  Y  

n Example:  How  good  is  your  registra7on  page?  
n Okay:  #  of  registered  users  per  day  
n BeUer:  registra7on  conversion  rate  =  
   #  registered  users  /  #  uniques  to  reg  page  
Copyright  ©  2014  Olsen  Solu7ons  
Registra7on  Page  Conversion  Rate  
Registration Page Conversion
Daily Signup Rate
Page Yield vs. vs. Time
Time
New Registered Users divided by Unique Visitors to Signup Page

100%

90%
Page Yield Rate

80%
Page Conversion

70%

60%

50%
Daily Signup

40%
Registration

30% Started requiring

registration

20%
Changed Added questions
messaging to signup page
10%

0%
1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1
0

Copyright  ©  2014  Olsen  Solu7ons  

View  Each  Business  Metric  as  a  Gauge  
Current  
Value  

Minimum   Maximum  
Possible   Possible  
Value   Value  

Copyright  ©  2014  Olsen  Solu7ons  

 Priori7zing  Product  Ideas  by  ROI  

?
Return  (Value  Created)  

4  
Idea  D  
3  
Idea  A   Idea  B  
2  
Idea  C  
 1  
Idea  F  

 1   2    3   4  
Investment  (developer-­‐weeks)  
Copyright  ©  2014  Olsen  Solu7ons  
 Iden7fying  the  “Cri7cal  Few”  Metrics  
n What  is  the  upside  poten7al  of  each  metric?  
n How  many  resources  will  it  take  to  “move  the  needle”?  
n Developer-­‐days,  7me,  money  
n How  much  will  the  needle  move?  Revenue  impact?  
n Which  metrics  have  highest  ROI  opportuni7es?  

Metric  A   Metric  B   Metric  C  

Good  ROI   Bad  ROI   Great  ROI  
Return  

Return  

Return  
Investment   Investment   Investment  

Copyright  ©  2014  Olsen  Solu7ons  

Case  Study  from  Intuit  

q Improving  UX  
q Improving  Business  Results  
-­‐>  Sign-­‐up  Conversion  Rate  

Copyright  ©  2014  Olsen  Solu7ons  

 Case  Study:  Account  Signup  Process  Redesign  

Abandonment Rate (7 Day Moving Average)

Steps 1-2
80%

70%
Abandonment Rate (7 Day Moving Average)

60%

50%

40%

30%

20%

10%

0%
10/14/02

10/21/02

10/28/02

11/11/02

11/18/02

11/25/02

12/16/02

12/23/02

12/30/02
10/7/02

11/4/02

12/2/02

12/9/02

1/6/03

1/13/03

1/20/03
Copyright  ©  2014  Olsen  Solu7ons  
 Analyzed  Drop-­‐Off  at  Each  Major  Sec7on  

100%
100%
Focus  on  
biggest  
80%
drop  
%  of  Users  

62.3%
58.8%
60%
50.9%

40% 34.4%
32.7%

20%

0%

Sign  in  /   Account  Type   Cash  vs.   5  Partner   3  Partner  

Registra7on   Margin   Pages   Pages  
Copyright  ©  2014  Olsen  Solu7ons  
 Analysis  of  Sign  In/Registra7on  Flow  
Open  
Account  
55%  
44%   Register   Registra7on   (24%  of  Total)  
  Process  

45%  drop  off  

64%  
(20%  of  total)  
of  Total  Account  
36%  overall   Selec7on  
83%   30%   drop  off  for  
56%   (46%  of  Total)   (14%  of  Total)   this  step  
Sign  in  
 

Forget   70%   Change   80%  

Password   (32%  of  Total)   Password   (26%  of  Total)  

17%  drop  off   20%  drop  off  

(10%  of  total)   (6%  of  total)  

Copyright  ©  2014  Olsen  Solu7ons  

 Redesigned  User  Flow  Improved  
Registra7on  Conversion  Rate  
Abandonment Rate (7 Day Moving Average)
Steps 1-2
80%

70%
Abandonment Rate (7 Day Moving Average)

60%

50%

40%

30%
Released 37% improvement
New Design in conversion rate
20%

10%

0%
10/14/02

10/21/02

10/28/02

11/11/02

11/18/02

11/25/02

12/16/02

12/23/02

12/30/02
10/7/02

11/4/02

12/2/02

12/9/02

1/6/03

1/13/03

1/20/03
Copyright  ©  2014  Olsen  Solu7ons  
Case  Study  from  Friendster  

q Improving  Business  Results  

-­‐>  Viral  New  User  Growth  

Copyright  ©  2014  Olsen  Solu7ons  

 Case  Study:  
Op7mizing  Friendster’s  Viral  Loop  
% of users
sending = 15% Invites per Invite
invites sender = 2.3 click-through rate

Active Invite Prospective Click Registration Fail

Users Users Process
% of users
who are Succeed
Don’t
active Click Conversion
= 85%
Users rate

•  Mul7plied  together,  these  metrics  determine  your  viral  ra7o  

•  Which  metric  has  highest  ROI  opportunity?  

Copyright  ©  2014  Olsen  Solu7ons  

 The  Upside  Poten7al  of  a  Metric  
?  

100%   100%  
85%  

15%   2.3  
0   0   0  
Registra7on   %  of  users  sending   Avg  #  of  invites  
Process  Yield   invita7ons   sent  per  sender  

Max  possible  
0.15  /  0.85  =  18%   0.85  /  0.15  =  570%   ?  /  2.3  =  ?%  
improvement  

Copyright  ©  2014  Olsen  Solu7ons  

 Okay,  so  how  can  we  improve  the  metric?  
n How  do  we  increase  the  average  number  of  
invites  being  sent  out  per  sender?  
n For  each  idea:  
n What’s  the  expected  benefit?  (how  much  will  it  
improve  the  metric?)  
n What’s  the  expected  cost?  (how  many  engineer-­‐
days  will  it  take?)  
n You  want  to  iden7fy  highest  ROI  idea  

Copyright  ©  2014  Olsen  Solu7ons  

Amer  Launching  Address  Book  Importer…  

Copyright  ©  2014  Olsen  Solu7ons  

Amer  Launching  Address  Book  Importer…  

Copyright  ©  2014  Olsen  Solu7ons  

Amer  Launching  Address  Book  Importer…  

Copyright  ©  2014  Olsen  Solu7ons  

If  you  could  only  track  1  metric  to  
measure  your  Product-­‐Market  Fit:  
Which  metric  would  it  be?  

Copyright  ©  2014  Olsen  Solu7ons  

 Reten7on  Rate  
n Reten7on  rate  tracks  what  %  of  your  
customers  are  s7ll  ac7ve  over  7me  

~80%
never use
app again

Curve
eventually
flattens out

Copyright  ©  2014  Olsen  Solu7ons  

Cohort  Analysis  

Copyright  ©  2014  Olsen  Solu7ons  

Cohort  Analysis:  Data  

Copyright  ©  2014  Olsen  Solu7ons  

Improving  Reten7on  Rate  Over  Time=  
Increasing  Product-­‐Market  Fit  

David  Skok,  Matrix  Partners  

hUp://www.forentrepreneurs.com/saas-­‐metrics-­‐2/  
Alternate  Ways  to  Track  Reten7on  
n Having  lots  of  cohort  curves  is  hard  to  read  
n Would  be  great  to  have  a  7me  series  metric  
=  one  metric  we  can  track  over  7me  
n %  Users  Retained  who  signed  up  X  days  ago  
n Can  use  single  or  mul7ple  X  (30  &  90  days)  

n Another  metric:  Returning  users  

n Good  summary  metric:  #  of  users  “locking  in”  
n Gives  a  sense  of  scale  (not  a  %)  
n Recommend  7-­‐day  average  (can  do  others  too)  

Copyright  ©  2014  Olsen  Solu7ons  

 Profitability,  anyone?  

Two  key  metrics:  

• Customer  Life7me  Value  (LTV)  
• Customer  Acquisi7on  Cost  (CAC)  
 
You  want:  
LTV  –  CAC  >  0  
Profitability,  anyone?  
 Profitability,  anyone?  

Two  key  metrics:  

• Customer  Life7me  Value  (LTV)  
• Customer  Acquisi7on  Cost  (CAC)  
 
You  want:  
LTV  –  CAC  >  0  
 Life7me  Value  (LTV)  
n Life7me  value  of  a  customer  =  how  much  value  
your  average  customer  will  generate  
n LTV  =  ARPU  x  Avg  Customer  Life7me  x  Gross  Margin  
n ARPU  (Avg  Revenue  /  User)  =  Total  Revenue  /  #  of  Users  
n Average  Customer  Life7me  
n How  long  your  average  customer  generates  revenue  
n Equals  1  /  churn  rate    (5%  monthly  churn  =  avg  life  20  months)  

n Gross  Margin:  the  %  of  revenues  lem  over  amer  

subtrac7ng  the  cost  of  providing  the  product/service  

Note:  for  simplicity,  this  LTV  equa7on  ignores  the  “cost  of  capital”  
Copyright  ©  2014  Olsen  Solu7ons  
 Customer  Acquisi7on  Cost  (CAC)  
n CAC  is  the  average  cost  for  you  to  obtain  a  
revenue-­‐genera7ng  customer  
n So  it  takes  into  account  both  your  cost  of  
acquiring  a  prospect  and  your  conversion  
rate  for  conver7ng  prospects  to  revenue-­‐
genera7ng  customers  
n CAC=Cost  per  Acquisi7on  /  Conversion  Rate  

Copyright  ©  2014  Olsen  Solu7ons  

What  You’d  Like  to  See  Over  Time  

n LTV  increasing  as  you  improve  your  value  

proposi7on,  customer  reten7on,  &  pricing  
n CAC  decreasing  as  you  op7mize  your  marke7ng:  
segments,  channels,  messaging  
Copyright  ©  2014  Olsen  Solu7ons  
 Ra7o  of  LTV  to  CAC:  
Real  data  from  HubSpot  

Copyright  ©  2014  Olsen  Solu7ons  

 Lean  Product  Analy7cs  Process  

Iden7fy  What  
Your    Metrics  Are  

Iden7fy  highest  
ROI  idea  
Measure  Metrics  
Baseline  Values  

Evaluate  Metrics   Select   Brainstorm   Learn   Design  and  

Ideas  to  
Upside  Poten7al   Top  Metric   &  Iterate   Implement  
Improve  Metric  

Analyze  How  
the  Metric  
Changes  
Global   Metric  
Level   Level  
Copyright  ©  2014  Olsen  Solu7ons  
Questions?
olsensolutions.com
linkedin.com/in/danolsen98
@danolsen

Copyright  ©  2014  Olsen  Solu7ons