2021-02-16 - Why A/B Testing

Is Likely Making Your UX Worse

On the surface, A/B testing and its multivariate kinsfolk feel like helpful tools. You
compare one or more new ideas to the existing idea in a controlled experiment,
and, using the scientific method, you can learn which idea produces the “best”
results.
However, under the surface, there lurk nefarious problems that not only guide
your team in the wrong direction but can create massive UX debt that will surprise
you later. It creates deep-rooted damage that, if not discovered early, can make it
almost impossible to deliver a great user experience in the not-so-distant future.
In this discussion, Jared will explore the deep-seated issues behind using
experimentation engines and conversion-rate optimization techniques. You’ll see
the damage it can do, and ways to mitigate the risks, while still getting benefits
from the techniques.

Before the talk gets started:

How this discussion will go:

20210216 - Why A/B Testing Is Likely Making Your UX Worse 1

 20 - 30 minutes short presentation on the today's topic.

Remainder of the time discussing your questions.

Please put your questions in the chat.

Summer Kohlhorst will collect the questions and pick the ones we'll
answer today.

Things to keep in mind with Zoom:

If the little Chat pop-ups get distracting, feel free to open the chat window.

Please put your microphone on mute.

If Summer chooses your question, you can unmute to talk about it with
Jared

Zoom's Speaker View will let you keep Jared visible and you'll see what
he's presenting on the screen.

This session is closed captioned.

We will post the recording for this session within a day.

Notes for today's discussion:

A/B testing is a type of experimentation.

It's a two-variant type of multivariate tests.

We take a variation and test it against a control.

We use a pre-determined goal to determine "success."

The variation that performs the best (as measured with the goal) is the
"winner."

Experiments involve a stimulus and a reaction.

20210216 - Why A/B Testing Is Likely Making Your UX Worse 2

 The stimulus is the change we make in the world.
The reaction is the change we see.

A hypothesis is a prediction of what the reaction will be to the stimulus.

We use observations to prove whether our hypothesis is correct.

We use inferences to rationalize why.

Observations are changes we can see in the world.

The observations are usually behaviors we directly witness.

Inferences are attributions as to why we think the changes occurred.

In a lot of experimentation, we confused observations and inferences.

Ideally, we turn inferences into hypotheses and then conduct a further

experimentation.
If we change this design, we should see a change in target behavior.

Often, however, people accept inferences as statements of fact.

This creates problems when we learn later they weren't true.

Definitions for UX measurement:

Measures: A change we can observe in the world.

Metrics: Measures we track.

KPIs: Important metrics.

Analytics: Measures computers can track.

A/B tests focus on analytics to determine "success."

Many analytics don't represent outcomes.

Instead they are measures of outputs.

20210216 - Why A/B Testing Is Likely Making Your UX Worse 3

 UX Metrics framework:
Based on UX Outcomes

If we do a great job on our design, how will we improve someone's life?

UX Success metrics measure the improvement we make in the user's life.

UX Progress metrics measure how close we are to success.

Problem-Value metrics measure how big a problem is.

Many of these can't easily be measured with analytics.

Just because a user can do something doesn't mean they should do that thing.

Just because we'd like a user to buy our product doesn't mean they should buy
our product.

Because not everyone should buy the product, we can't expect a 100%
conversion rate.
However, without knowing how many folks should buy, we can't calculate
the top possible rate.

This means we don't know what our success goal should be.

A/B testing requires us to infer as to the cause.

We don't know why one solution beat the other.

Not knowing why isn't a complete issue, but it requires us to be more robust.
We need a lot more data to have accurate predictions of outcomes.

We almost never check why it failed for some users.

A/B testing requires homogeneous participation.

However, our participation is almost always heterogeneous.

It's a forced between-subjects experiment.

We can't tell if the differences between one variant is because of the variations
or the sampling.

20210216 - Why A/B Testing Is Likely Making Your UX Worse 4

 A/B testing also requires us to reduce the stimulus to a single variable.

We may not realize that we've failed at this.

We may be changing multiple things at a time.

A/B testing is subject to local maxima distortions.

This is when we think we've hit the "best we can do", but miss an entire line of
thinking in design variation.

Qualitative testing needs to drive our experimentation tests.

We need to focus more on outcomes than just outputs.

We need to understand why a variation performs better before blindly
accepting it.

We use qualitative studies to understand the problem.

Within the problem, we should find hypotheses that prove reliable.

Our UX strategy needs to be driven primarily by qualitative research.

Quantitative experimentation can prove effective, once we understand why

we're seeing what we seeing.

Upcoming Talk UX Strategy topics:

February 22, 2021 Designing for Delight Is the Ultimate End-Goal

March 1, 2021 Getting the Most From Our Subject-Matter Experts

Resources for this week's topic:

20210216 - Why A/B Testing Is Likely Making Your UX Worse 5

 Is Design Metrically Opposed?

UIE
In this presentation, Jared shares: What easily-collected
analytics, like bounce rate and time-on-page, actually tell us
about our users' experiences Why advanced techniques, like a
https://www.uie.com/designmetrics/

20210216 - Why A/B Testing Is Likely Making Your UX Worse 6