WH
VIDEO
Analyzing an Experiment: Etch A Sketch
Analyzing an Experiment: Etch A Sketch
Analyzing an Experiment: Etch A Sketch
Analyzing an Experiment: Etch A Sketch
Analyzing an Experiment: Etch A Sketch
Analyzing an Experiment: Etch A Sketch
Analyzing an Experiment: Etch A Sketch
STEPS
Calculating whether it's worth it to run an ad campaign
Step1: calculate market share for test
and control = test units/(test units +
control units) Pre-test
Test
Step 2: calculate lift for test and
control = test market shares/pre-test
market shares Lift
Step 3: calculate net lift = test lift –
control lift Net lift
Retail price
Retail margin
Step 4 Calculate manufacturer selling
price = price * (1-retail margin) Manufacturer selling price
Manufacturer contribution margin %
Step 5 Calculate contribution margin $
= manufacturer contribution margin %
* manufacturer selling price Manufacturer contribution margin $
National budget
Step 6: Calculate break even units = nat Units break even
Base units
Base units test period
Step 7: Calculate break even lift as a % Break even lift % of base
of the base (Base units are given, not
calculated) = units break even/base
units test period

Analyzing an Experiment: Etch A Sketch
Projecting lift from test to national to all sales to selling season (no pre-test sales to compare to)
Analyzing an Experiment: Betty Spaghetty
Analyzing an Experiment: Betty Spaghetty
Projecting Lift
Calculating Projected Lift: Betty Spaghetty
Calculating Projected Lift: Betty Spaghetty
Calculating Projected Lift: Betty Spaghetty
Calculating Projected Lift: Betty Spaghetty
Calculating Projected Lift: Betty Spaghetty
Step 8: Is it worth it? = only if net lift is
greater than break even lift % of base
Step 1: Calculate lift = test product
sales/control product sales
Step 2: Calculate break even units =
budget/manufacturer contribution
margin ($)
Step 3: Calculate projected national
retailer sales
Step 4: Calculated projected national
units sold
Step 5: Calculate projected annual
sales
Step 6: Calculate projected base sales Holiday units without ads (ak
Step 7: Calculate projected units after
ads during the period
Test weeks
Net lift
Ad budget
Retail selling price
Retail margin %
Manufacturer suggest price
Manufacturer contribution ma
Manufacturer contribution ma
Break even units
Test % of control (CA) sales
Total control (CA) units sold
Control % of national sales
National retailer sales
Retailer share of product sale
National units sold
Test % of annual sales
Annual sales
Holiday % of annual sales
Lift from ads
Units after ads in holiday sea
Calculating Projected Lift: Betty Spaghetty Step 8: Compare to break even units Break even units

Running a web experiment

Maximizing Effectiveness: Nanoblocks

Eiffel Tower "Gold b
Taj Mahal
Neuschwanstein
Etch A Sketch
 WHAT'S SHOWN IN VIDEOS

th it to run an ad campaign
Test Product: Etch A Sketch Control Product: Doodle

Cincinatti Cincinatti
Number (Test) Control* Cincinatti (Test) Control Cincinatti
of weeks Units Units Shares (%) Units Units Share

12 162 1526 9.6% 1517 6742 18.4%

3 240 1598 13.1% 816 3780 17.8%

136.1% 96.7%

39.4%

$15
40%

cturer selling price $9

cturer contribution margin % 0.53

acturer contribution margin $ $5

1,750,000
366,876
5,300,000
nits test period 2,333,000
ven lift % of base

15.73%
 TEST CONTROL

Arizona California California

Color Arizona Go Color Go Go
Crazy Go Glam Crazy Glam

4 1.8 2.2 0.3 1.2

267%

3000000
15
36%
cturer suggest price 9.6
cturer contribution ma 58%
cturer contribution ma 5.568
538793

of control (CA) sales 10

ontrol (CA) units sold 1420
% of national sales 12.0%

al retailer sales 11833

share of product sale 25.0%

47333
of annual sales 5.50%

860606
% of annual sales 45.00%

units without ads (ak 387273

267%

fter ads in holiday sea 1032727

 538,793

Lift on
units sold
promotion
vs. pre-
Sales priceJan - Feb Mar May promotion
$19.99 274 686 219 501%
$19.99 308 163 132 106%
$19.99 244 184 146 151%
$19.99 344 352 399 205%