Lean Analytics

Lean Product & Lean UX Silicon Valley - July 2014

Hiten Shah
hnshah@gmail.com

Download the slides: h!p://kiss.ly/leanprod2014

Continuous Learning & Improvement
Don’t know where to start?
Wondering if you’re working on the right things?
Or, you just want to hear me babble!
HTTP://KISS.LY/LEANLYTICSBOOK
Lean Analytics helps you Optimize for Learning
Lean Analytics Cycle

AVINASH KAUSHIK HTTP://KISS.LY/LEANAC

Metrics  →  Hypothesis  →  Experiment  →  Act
HTTP://HELLOBAR.COM
What is Hello Bar?
  Metrics  
 Figure  out  what  to  improve  

 Hypothesis  
 Make  an  educated  guess  
   
 Experiment  
 Test  your  guess  

 Act  
 Decide  what’s  next
Our funnel showed a low installation rate

Hello Bar Funnel Metrics

Steps Visitors Homepage Created Completed Installed
URL Entries First Bar Registration
Totals 15068 1868 716 688 398
% prev step 12.40% 38.33% 96.09% 57.85%
% drop off -87.60% -61.67% -3.91% -42.15%
% of visitors 100% 12.40% 4.75% 4.57% 2.64%
The Lean Analytics Cycle At Work

We  achieved  a  40%  increase  for  our  

installa2on  rate  with  just  this  one  
first  experiment.
Original Hello Bar installation screen
  Metrics  
 Figure  out  what  to  improve  

 Hypothesis  
 Make  an  educated  guess  
   
 Experiment  
 Test  your  guess  

 Act  
 Decide  what’s  next
Find out why people don’t install Hello Bar
Didn’t Meet My Expectations
I Had Trouble Installing It
I Don’t Have A Website
Not Ready To Install
Other
Hello Bar Experiment Hypothesis

If  more  installa2on  op2ons  are  given,  

then  more  people  will  add  Hello  Bar  
to  their  website  due  to  the  need  for  a  
WordPress  plugin  and  ability  to  email  
the  code  to  someone  else.
  Metrics  
 Figure  out  what  to  improve  

 Hypothesis  
 Make  an  educated  guess  
   
 Experiment  
 Test  your  guess  

 Act  
 Decide  what’s  next
Created a new flow to test
I can install code myself.
41.67% responded with “WordPress Plugin”
25% responded with “email my developer / web designer”
I need help.
  Metrics  
 Figure  out  what  to  improve  

 Hypothesis  
 Make  an  educated  guess  
   
 Experiment  
 Test  your  guess  

 Act  
 Decide  what’s  next
Full funnel metrics for the experiment

Control
Steps Visitors Homepage Created Completed Installed
URL Entries First Bar Registration
Totals 7469 925 353 341 197
% prev step 12.38% 38.16% 96.60% 57.77%
% drop off -87.62% -61.84% -3.40% -42.23%
% of visitors 100% 12.38% 4.73% 4.57% 2.64%

Variation
Steps Visitors Homepage Created Completed Installed
URL Entries First Bar Registration
Totals 7456 906 342 329 276
% prev step 12.15% 37.75% 96.20% 83.89%
% drop off -87.85% -62.25% -3.80% -16.11%
% of visitors 100% 12.15% 4.59% 4.41% 3.70%
Resulted in aresulted
Experiment 30% installation
in a 40% improvement
rate increase

HTTP://ISVALID.ORG
The Lean Analytics Cycle At Work

ACer  a  dozen  more  experiments,  our  

installa2on  rate  is  now  hovering  
between  5%  and  7%  

That’s  an  addi;onal  89%  increase!!!

How You Can Put It Into Practice

Metrics  →  Hypothesis  →  Experiment  →  Act

 Metrics  
Figure  out  what  to  improve
“What gets measured, gets managed.” Peter Drucker
Baselines

DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA

Funnels

KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

SaaS Funnel

KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

eCommerce Funnel

KISSMETRICS FUNNEL REPORT HTTP://KISS.LY/KMFUNNEL

Revenue

BUFFER BAREMETRICS HTTP://KISS.LY/BUFFERBARE

Spreadsheets and Calculations

Steps Visitors Homepage Created Completed Installed Created

URL Entries First Bar Registration Second Bar
Totals 15068 1868 716 688 398 193
% prev step 12.40% 38.33% 96.09% 57.85% 48.49%
% drop off -87.60% -61.67% -3.91% -42.15% -51.51%
% of visitors 100% 12.40% 4.75% 4.57% 2.64% 1.28%

HELLOBAR HTTP://HELLOBAR.COM
Hypothesis  
Make  an  educated  guess
“A solid test hypothesis is an informed solution to
a real problem – not an arbitrary guess. The more
research and data you have to base your
hypothesis on, the be!er it will be.”


— Michael Aagaard, h!p://kiss.ly/abthypo

 Research  
Tools  &  Tac;cs
Customer Development

HTTP://WWW.CUSTDEV.COM
Jobs To Be Done #JTBD

HTTP://JOBSTOBEDONE.ORG
Usability Testing

HTTP://USERTESTING.COM
Live Chat

HTTP://OLARK.COM
Mouse Tracking

HTTP://CRAZYEGG.COM
Long Form Surveying

HTTP://SURVEYMONKEY.COM
On-Page Surveying

HTTP://QUALAROO.COM
Personalized Messaging

HTTP://INTERCOM.IO
Email a survey to people

HTTP://SURVEYMONKEY.COM
Ask people why they canceled

HTTP://SURVEYMONKEY.COM
Get feedback right when they cancel
Exit Surveys

HTTP://QUALAROO.COM
Discover what’s missing

HTTP://QUALAROO.COM
Find out what words people use to describe your product

HTTP://QUALAROO.COM
Learn about people’s expectations

HTTP://INTERCOM.IO
Learn what frustrates people

HTTP://INTERCOM.IO
More actionable reading…

QUALITATIVE DATA HTTP://KISS.LY/QUALABT

Experiment Hypothesis Formula

DATA DNA EBOOK HTTP://KISS.LY/OPTDATADNA

Experiment  
Test  your  guess
Only  1  out  of  5  tests  win.

4  out  of  5  tests  

cost  you  money.

$
Answer key questions for each experiment

1)  Which  top  metric  are  we  trying  to  improve?  

2)  What  can  you  anchor  your  metric  against?  

3)  What  is  your  hypothesis  about  why  this  metric  is  at  x%?  

4)  What  do  we  expect  from  this  experiment?  

5)  What  was  the  result  and  what  did  we  learn  from  it?

HTTP://KISS.LY/6QSAB
Prioritize your experiments

HTTP://KISS.LY/ABPIE
Don’t copy “best practices” blindly
Learn what works best for you
 Act  
Decide  what’s  next
 Explore  the  results  and  make  
data  informed  decisions.

THE ANALYTICS THAT MATTER TO FACEBOOK HTTP://KISS.LY/FBINFORM

KISSmetrics A/B Testing Report

KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

We know that people love Google Analytics
KISSmetrics Experiment Hypothesis

If  people  authen2cate  with  Google,  

then  more  people  will  sign  up  due  to  
their  exis2ng  familiarity  with  Google  
Analy2cs  and  alignment  with  our  
winning  headline.
KISSmetrics Homepage Test Control
KISSmetrics Homepage Test Variation
KISSmetrics Homepage Test Signup Conversion Rate

KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

KISSmetrics Homepage Test Received Data

KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

KISSmetrics Homepage New Control
Inspiration
KISSmetrics Homepage New Test Variation
KISSmetrics Homepage Test Signup Conversion Rate

KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

KISSmetrics Homepage Test Received Data

KISSMETRICS A/B TESTING REPORT HTTP://KISS.LY/KMABTEST

Growth!

HTTP://KISSMETRICS.COM
  Metrics  
 Figure  out  what  to  improve  

 Hypothesis  
 Make  an  educated  guess  
   
 Experiment  
 Test  your  guess  

 Act  
 Decide  what’s  next
 Hiten Shah
hnshah@gmail.com
@hnshah

You will get all you want in life if you help enough
other people get what they want.
Zig Ziglar

Download the slides: h!p://kiss.ly/leanprod2014