GoodUI DATASTORIES - ISSUE #1 - MAY 2014

GoodUI DATASTORIES - ISSUE #1

Our Very First Test

The first A/B test you do can be quite simple - just the control versus one B variation. That’s exactly what
we did as we ventured out onto the road of conversion optimization less than a year ago. The goal was
defined clearly by the client who wished to squeeze Quotes Completed on a form page. Here is what
and how we did it. Hoping you can learn from it.

The Overview
Test ran for 61 days with the Control and 1 Variation

A B

Measuring Quotes
Completed

+53.5%
Relative Improvement

Control Best Variation

Converted at 22.25% with 89/400 visits Converted at 34.15% with 139/407 visits

The Test Setup

Visitor Types Tool Used Page Scope
100% of All Traffic Visual Website Optimizer Single Quote Page

Test Duration Design & Build Time # of Variations

61 Days 2-3 weeks One

Primary Metric: Quotes Completed Test Type

Tracked quotes completed with mouse clicks on button Split URL Test

1 HTML creation with a distinct URL

Due to the many differences in the new B variation, we decided to create a separate HTML file
for it. Visual Website Optimizer also has an inline HTML editor that can modify the HTML
against the control, however it’s better suited for smaller changes.

2 CSS class attachment

In order to track the actual quotes completed, a custom CSS class was made up and assigned
to the <A> tag within both the control and the variation. Here is the simple code:

<a class="ourButton" href="javascript:submitPage();">Request A Quote</a>

3 Test Creation inside VWO

Within Visual Website Optimizer, we then created a new Split URL Test which would handle the
randomization and automatic splitting of traffic for us. To enable the actual URL splitting, VWO
provided a JS code snippet to be placed on the control page just before the </head> tag - easy.

4 Goal tracking inside VWO

Within VWO we then created a goal to be triggered when a user “clicks on an element”. Here
we referenced the “.ourButton” class and started the test.

A The Control
This is what the standalone insurance quote page looked like which we were competing against. It was a
rather subtle form that was shifted to the right side. The form also contained small sized fields with a small
call to action. A few positive things about it which we were going to keep were: social proof, communica-
tion of clear benefits/reasons for signing up and a strong invitation to join.

B The Variation
There was one variation in this experiment and we put everything we knew about form design into it. We
hypothesized that users were not noticing the form. Hence, we made the variation more visible
while also removing any resistances to filling it out. Listed below are the key improvements which we
believe differed from the control:

Social Proof
idea

4
Three testimonials were used in this variation as opposed to just one in the control.

GoodUI Being Direct

idea

Ideas 10
We used language that instructed the user very clearly to “Get a Quote”.

Fewer Form Fields

idea

13
The following fields were removed: Zip, County, Current Policy Expiration and Phone Number

Keeping Focus
idea

16
Floating social media buttons were removed to keep the attention focused on the primary task.

Benefit Button
idea

18
As opposed to using a generic “Submit” button label, the “Request A Quote” button was used.

Bigger Click Areas

idea

38
Button and input field sizes were increased for easier clickability.

Additional
Ideas Focused Fields
All fields had a clearly glowing “selected” state and the first field was autofocused.

Central Form
The form was placed centrally, took up more space, and had a higher visual contrast. More so, the
First Name was made larger to grab more attention.

The Results

B generated +53.5% more quotes

The 95% confidence interval was between +25% and +79%

In absolute terms, that’s a change in quotes from 22.25% to 34.15%

Cumulative Effect Graph For The Primary Metric

50%
40%
30%
20%

Day 1 Day 61

After 61 days of waiting, a clear winner emerged - variation B. Going back to our hypothesis and looking at
the changes in the variation, we think that the improvement was caused by two simple and fundamental
strategies. First of all, we gave users a bit of an extra push by making the the form more visible, while
making the wording more direct. Secondly, we also removed some of the friction or obstacles to task
completion with fewer and larger form fields. Triggering the customer to act more prominently while
decreasing cognitive friction, have potentially caused the lift.

The Decision
At the end of the day, one has to make a call whether to implement the new variation or not.
The reason why we recommended to the client to implement version B was due to our confi-
dence in the improvement. We mostly used the margin of error from the relative increase to
gauge this. With a 95% confidence we calculated that the relative increase must have been
somewhere between +25% and +79%. This told us that the variation was clear of the possibility
of being a losing one. Even though the margin of error was quite wide, we were still pretty sure
that it was an improvement nevertheless. So the decision to implement was made.

Looking Back & Key Learnings

More Variations
The one thing that we changed right away in our process following this test is that we now attempt to do
more variations per each test. In retrospect, having just one shot at the results was a bit risky. Perhaps in
our first test we were confident, or we picked some obviously low hanging fruit, or we just got lucky -
maybe a combination of all. Nevertheless, although more variations add to test duration, effort, and a
chance for false positives, they do lower the overall risk. So while running tests, we do recommend to
strive for more variation diversity.

Measuring Deeper Metrics

On this test we measured clicks on a submit button, but this metric could have been complemented by a
deeper (and possibly more accurate) metric from reaching a goal page. Visual Website Optimizer allows to
to trigger a conversion that can fire off when customers reach a particular page deeper in the funnel. This
can remove some noise from the initial click counts that could have fired with improper data or missing
fields. Deeper metrics can be used to remove this problem.

Additional Variation Ideas

Below are a few additional ideas we’d be eager to test if we would run the test again.

C Further attention grabbing with animation to the first field.

D Further increasing the size of the fields.

E Further decreasing the number of form fields by merging the first name and last name into a
single name field. (Could be split with Javascript before sending to server)

F Inline validation and positive reinforcement for each completed field.

In The Next Issue

Next month we might look at a test which was more cyclical while benefitting from a retest.
If you’d like to see a particular test, have an idea or comments about this issue, please send your emails to jakub@linowski.ca.

Thank you again for supporting the project!

GoodUI DATASTORIES - ISSUE #1 - MAY 2014