Point Estimate for the

difference between Control

(A) and Treatment (B) Confidence

Confidence Interval for

Interval 1
the difference

Practical Significance
Boundaries, based on
business
2
considerations Point Estimate for
the difference

Practical Significance 0 Practical Significance

Boundary Boundary
1 A (relatively) easy decision

Confidence Interval

Point Estimate

Practical Significance
Boundary
0 Practical Significance
Boundary

The Confidence Interval

Statistical Significance: does not include 0

The Confidence Interval is

Practical Significance: outside of the Practical
Significance Boundaries

Decision: Launch
 2 The change has no impact

Confidence Interval

Point Estimate

Practical Significance
Boundary
0 Practical Significance
Boundary

The Confidence
Statistical Significance: Interval includes 0

The Confidence Interval stays

Practical Significance: inside of the Practical
Significance Boundaries

Decision: Iterate or abandon the idea

3 Confident estimate but little
practical relevance
Confidence Interval

Point Estimate

Practical Significance
Boundary
0 Practical Significance
Boundary

Confident about the

Statistical Significance: magnitude of change...

...but that magnitude might not

Practical Significance: be sufficient to outweigh costs

Decision: Probably not worth launching

 4 Not enough data
Confidence Interval

Point
Estimate

Practical Significance
Boundary 0 Practical Significance
Boundary

We don't have enough

Statistical Significance: power to draw strong
conclusions...
...but the confidence interval is
Practical Significance: outside of what is practically
significant

Decision: Run a follow-up test with more

units
 5 Not enough precision
Confidence Interval

Point Estimate

Practical Significance
Boundary
0 Practical Significance
Boundary

The estimate is not

Statistical Significance: precise enough...

...even though the result could

Practical Significance: be practically significant

Decision: Repeat the test with greater

power
 6 Repeat if you have time
Confidence Interval

Point Estimate

Practical Significance
Boundary 0 Practical Significance
Boundary

The result is statistically

Statistical Significance: significant...

...and likely (but just "likely")

Practical Significance: practically significant

Decision: Repeat the test with more

power. If you have to decide, launch
 I share my learnings in
growth hacking and growth
management, talking about
the intersection between
Product, Marketing and
Experimentation