GoodUI Data Stories Issue 5

GoodUI DATASTORIES - ISSUE #5 - SEPTEMBER 2014
GoodUI DATASTORIES - ISSUE #5
When Words & Forms Collide

Online forms are typically ordered in a top to bottom fashion combined with some sort of short labels here
and there. Copy is minimal as input fields take center stage. The online sweepstakes site Prizegrab.com
followed this exact convention as we began working together to Raise Contest Signups on one of their
prize pages. Granted, that some vertical flow does help the reader jump from one field to the next as they
fill everything out. Bored with convention however, we took this as an opportunity to explore the so called
idea of Natural Language Forms. We wanted to find out if form fields could be fused together with words
and sentences for an even stronger conversion effect. Would these form-in-sentence combinations convert
better or would they just become lost or ignored? Here is what we found ...
Overview
The test was run across 13 days with the Control and 6 Variations
A E
+28%
To Signup Clicks
Measuring
Contest
Signups
Best Variation
Control Converted at 17.4% with 283 out of 1626 visits
Converted at 13.6% with 214 out of 1574 visits
Key Causes Of Effect

idea
48 Natural Language
The text was written in first person narrative while merging form fields with sentences.
idea
13 Fewer Form Fields
First & last names were merged into one field, the duplicate password confirmation was removed,
and the personal birthday field was changed into a confirmation text of being “18 years or older”.
idea
50 Reassurances
The addition of the sentence reading “and I want to win $1,000, because a Winner is Guaran-
teed!” carries with it a reassuring guarantee.
idea
24 Selling Benefits
teed!” reminds of the benefit of winning a valuable prize.
Higher Up Button
The “Continue” button moved higher up removing the chance of being hidden below the fold.
Input Styling
The input field was styled as a link with an underline.
One Less Subheader

The “Official Entry Form” text was removed.
Test Snapshot
Test Duration & Timeframe
End of Test
Our safe bet variation A
C
Variations
D All variations reached

the end of the test.
None were removed
E during testing.
Our best performer
F
The Control Insignificant Variations (p-Value higher than 0.2)

Significant Winners (p-Value less than 0.05) Almost Significant Winners (p-Value less than 0.05)
Significant Losers (p-Value between 0.2 and 0.05) Almost Significant Losers (p-Value between 0.2 and 0.05)
A The Control
This was the original prize signup page. Traffic which arrived here was coming from an email newsletter as
well as organically from the main homepage. Some of the positive things already applied on this page in
terms of conversion rate, included:
idea
24
Selling
Benefits
idea
11
Good Contrast
Existing Elements Which We Think Worked Well:

idea
Read more on
these ideas at 11 Good Contrast
GoodUI.org
Compared to everything else, the “Continue” button was visible and stood out.
idea
Who doesn’t like a chance at winning $1,000. Can’t argue with that benefit being clear.
B The B Variation & Its Effects
Variation B was our “safe bet” idea containing the lowest risk and highest certainty for improvement.
Knowing from past tests that “Fewer Form Fields” is an easy win, it was incorporated into variation B and
others throughout the test. In case all else failed, at least we would be able to achieve a quick win
improvement nevertheless. Notice that it did not contain anything related to the idea of “Natural
Language” as the form fields were organized traditionally, top to bottom. Results are grey if
they aren’t conclu-
sive (p-Value is
Compared To A +13% higher than 0.20).

Contest Signups
idea
13
Fewer Form
Fields
The Control
Higher Up
Button
Differences Which We Think Have Contributed To The Effect:

idea
Higher Up Button
Interesting Comparison
B F
+13% +19%
Contest Signups Contest Signups
Converted at 15.3% with 240 out of 1568 visits Converted at 16.2% with 260 out of 1603 visits
C The C Variation & Its Effects
Variation C was the first to introduce the idea of “Natural Language” into the test. Here we crafted a few
sentences which were written from the perspective of the reader (in first person). All fields were then
merged into these sentences which we believe justified the reason for their asking. Results are green if
they are conclusive
(p-Value is less than
Compared To A +27% 0.05). Nice!

Contest Signups
idea
48
Natural
Language
idea idea
13 50
Fewer Form Selling
Fields Benefits
idea
50
Reassurances
The Control
Higher Up
Button

idea
48 Natural Language
idea
idea
50 Reassurances
idea
Higher Up Button
D The D Variation & Its Effects
Variation D tried to improve upon C in making the input fields more visible with higher contrast and addi-
tional attention grabbing animations. As we began to speculate that fields embedded into copy might get
lost, this variation tried to combat this problem.
Results are grey if
they aren’t conclu-
sive (p-Value is
Compared To A +14% higher than 0.20).

Contest Signups
idea
One Less 48
Subheader
Natural
Language
idea idea
13 50
Fewer Form Selling
Fields Benefits
idea
50
Reassurances
The Control
Higher Up
Button
Additional
Interactions
Interaction 01: When the page first loads, the dark field background
color would fade in and out twice and then stop, for additional attention.
Interaction 02: Selected state with full white background field color.

idea
48 Natural Language
idea
idea
50 Reassurances
idea
Higher Up Button
Additional Interactions
Attention grab fades and hover states were introduced.
One Less Subheader

E
Be
The E Variation & Its Effects
st
Pe
rfo
rm
er
Variation E took a different approach to the speculation of “less visible” input fields. It was very similar to
D with the exception of input fields looking like text links with wider underlines. We also removed one
additional subheader to maintain focus.
Compared To A +28%
Contest Signups
idea
One Less 48
Subheader
Natural
Language
idea idea
13 50
Fewer Form Selling
Fields Benefits
idea
50
Reassurances
The Control
Input Styling
Higher Up
Button

idea
48 Natural Language
idea
idea
50 Reassurances
idea
Higher Up Button
Input Styling
One Less Subheader

Interesting Comparison
D E
+14% +28%
Contest Signups Contest Signups
Converted at 15.3% with 251 out of 1616 visits Converted at 17.4% with 283 out of 1626 visits
F The F Variation & Its Effects
Variation F was created by taking E and removing one additional sentence in the form paragraph. This was
the sentence which reassured about winning as well as the benefit.
Results are light green
if they are somewhat
conclusive (p-Value is
Compared To A +19% between 0.2 and 0.05).

Contest Signups
idea
One Less 48
Subheader
Natural
Language
idea
13
Fewer Form
Fields
The Control
Input Styling
Higher Up
Button

idea
48 Natural Language
idea
Higher Up Button
Input Styling
One Less Subheader

G The G Variation & Its Effects
Variation G was exactly the same as E except with larger text. Again we were trying to increase the visibil-
ity and importance of the key paragraph containing the input fields. As a result of the increased font size,
the call to action button shifted lower. Results are light green
if they are somewhat
conclusive (p-Value is
Compared To A +21% between 0.2 and 0.05).

Contest Signups
idea
One Less 48
Subheader
Natural
Language
idea
13 idea
50
Fewer Form
Fields Selling
Benefits
idea
Larger Text 50
Reassurances
The Control
Input Styling

idea
48 Natural Language
idea
idea
50 Reassurances
idea
Input Styling
One Less Subheader

Larger Text
The text of the form paragraph was enlarged.
Test Setup & Results
This was somewhat of a Little Test. The number of changes across the variations were kept rather small as
to allow easier cross comparison and identification of cause of effects. Overall we wanted to learn if the
introduction of natural language forms would have a positive effect on the signup rate.
The test was setup on a dynamic page where a user would first make a choice to join the contest with
either their Facebook account or with an email address. We only focused on the email / account flow as it
was the one which had more fields to deal with (whereas the Facebook flow was simple and we didn’t have
much control over). Also most people signed up with their email accounts.
This time around, Optimizely was used as the tool for testing even though it was our first time using it. The
client insisted on having all of the data in their account for future reference - fair enough. We also thought
it would be interesting to try something new and learn from it.
Test Setup
Visitor Types Tool Used Page Scope

100% of All Traffic Optimizely.com Single Page
Test Duration Design & Build Time # of Variations

13 Days (from Aug 2014) 2 weeks Control + Six
Primary Metric: Contest Signups Test Type

Tracked with clicks on main button A/B Test
Variations are grey if

Results of Primary Metric: Contest Signups (Clicks)
they are not conclu-
Tracked with clicks on main button.
sive (p-Value is
higher than 0.20). We’re 95% sure the
A Converted at 13.6% (11% to 16%) with 214 out of 1574 visits p-Value Improvement effect is somewhere
within this range.
B Converted at 15.3% (13% to 18%) with 240 out of 1568 visits 0.57 +13% (-12% to 36)
Variations are green
if they are conclusive
C Converted at 17.3% (15% to 20%) with 271 out of 1565 visits 0.021 +27% (2.2% to 52%)
(p-Value is less than
0.05). D Converted at 15.5% (13% to 18%) with 251 out of 1616 visits 0.43 +14% (-10% to 38%)
E Converted at 17.4% (15% to 20%) with 283 out of 1626 visits 0.015 +28% (3% to 52%)
F Converted at 16.2% (14% to 19%) with 260 out of 1603 visits 0.16 +19% (-5.3% to 43%)
Bonferroni Correction G Converted at 16.4% (14% to 19%) with 263 out of 1602 visits 0.12 +21% (-3.9% to 45%)
adjusts for false A lower p-Value
positives by increas- Data is shown at a 95% confidence interval and adjusted for multiple variations using the Bonferroni correction.
suggests stronger
ing the p-Value with significance.
each new variation.
The test was stopped in early September when the prize contest reached the end. We still managed to
reach significant results nevertheless. Here is what we found:
FINDING Natural Language Works

There are two interesting comparisons which show that using a Natural Language pattern has a positive
effect on the signup rate. We’ll compare to B instead of the control as B was our “adjusted control” with
the same number of fields as E.
First, looking at B (with traditionally ordered fields) it converted at 15.3%, whereas E (using natural
language) converted at 17.4%. Variation E simply reordered the input fields into a sentence form written
from the perspective of the reader. As part of the natural language paragraph, we also extended the
second sentence to read “and I want to win $1,000, because a Winner is Guaranteed!”. We felt that this
was a powerful way of motivating people during form entry as it provided a reassurance and benefit.
Secondly, slightly weaker but still showing an improvement, we can compare F to B. Variation F was
exactly the same as E, however it did not contain the additional “benefit and reassuring” sentence. This
made F was even more similar to B. Variation B converted at 15.3% and F and 16.2%. This tells us that the
basic ordering of fields into a paragraph form and being written in first person still maintained the positive
effect.
FINDING Input Field Styles Matter

We had three input fields styles which we tried out. These included: traditional input fields (C converted at
17.3%), custom dark grey backgrounds with animation (D converted at 15.5%) and text links with under-
lines (E converted at 17.4%). From this we can gain a hint that using custom white-on-dark background
colors with fade-in animations isn’t as good as the other two styles. Aside of these styles, everything else
on these three variations was the same, (except C also had an extra “Official Entry Form” subheading).
FINDING
The Fewer Fields The Better
This probably isn’t anything new however, comparing B to A there was a hint that fewer form fields
performed somewhat better. Variation A with an extra password field, a birthday field, and two fields for a
name (instead of a single full name) converted at 13.6%. Variation B without all of those extraneous fields
converted at 15.3%. The evidence for this isn’t strong, but it’s something we’ve seen in previous tests and
so this is just a reminder of the cause and effect
The Decision
It’s business as usual. We recommended to take the highest performer - variation E. The
E positive effect was +28% to Contest Signups with a p-Value of 0.015. We were pretty
confident with this.
Looking Back & Key Learnings

Stacking Variations As a Testing Strategy
Something which we began practicing this time around was what we’ll call Stacking Variations or a Serial
Test. The idea is simple. We introduce small changes gradually across the variations and have them build
on top of each other as we move from variation B forward. Hence, C builds on top of B, and D builds on top
of C, etc. Although we began doing this here, I think we could have adhered to this pattern more strictly.
(Sometimes we introduced more than one change and at other times we ignored this pattern).
Having a “Safe Bet” Variation

We formally made sure that the first variation was a fail safe one in case the rest of the natural language
variations (C to G) did not work. Therefore, although variation B had nothing to do with our hypothesis, it
was a way to provide the client with at least some positive result (which we felt it would) had everything
else failed. Doing this “defensive testing” which combines lower and higher risk variations could be good
practice for future tests.
It’s Good Collect Multiple Metrics

Notice that we have used “Clicks” as our primary metric, whereas we often suggest to use “Page Visits”
for more reliability. Well, we did have multiple page visit metrics setup along with a few other things.
However since it was our first time using Optimizely, some of the page visit data we had to ignore due to it
being distorted. For the control, our page visits to a thank you page fluctuated at around 70%, whereas our
clicks to contest signup was at around 13%. There is no way the site could have had a 70% conversion rate.
We almost thought we had to scrap the whole test until we found a good reason to trust the clicks data.
During an initial kick off session with the client, we did learn that a typical conversion rate was around
20% or so (for Facebook and Email combined). Luckly enough we also collected Facebook clicks. When we
added the measured Facebook clicks to the Email signup clicks, they turned out to be around 20% which
matched our expected conversion rate for the control. With this we are able to use the clicks data as a
reliable source for the test.
Additional Variation Ideas

Going forward, if we were to run more tests here are some variations which we might do:
H Top Aligned Fields With Explanations
It would be interesting to test and see if the same effect on natural language forms can be
duplicated without merging the fields into sentences, but instead showing the reasons for
asking them to the right side of the form fields.
I Fewer Distracting Links (Official Entry Rules)
It also might be interesting to completely remove the official entry rules link from the homep-
age and see if that hurts or helps conversion.
J Inline Entry Rules
Alternatively, the official entry rules could also be summarized as bullet points or explained in
plain language on the main page.
If you’d like to see a particular test, have an idea or comments about this issue, please send your emails to jakub@linowski.ca.
Thank you again for supporting the project! We hope you enjoyed the issue and learned from our insights. :)
Also a big thanks to Visual Website Optimizer, www.VWO.com, for supporting us by providing Datastories with an awesome a/b testing tool.
LINOWSKI
INTERACTION DESIGN
GoodUI DATASTORIES - ISSUE #5 - SEPTEMBER 2014

Appendix: Screenshots
A - The Control
E Variation

GoodUI Data Stories Issue 5

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

GoodUI Data Stories Issue 5

Uploaded by

Copyright:

Available Formats

GoodUI DATASTORIES - ISSUE #5 - SEPTEMBER 2014

GoodUI DATASTORIES - ISSUE #5

When Words & Forms Collide

Key Causes Of Eﬀect

One Less Subheader

D All variations reached

The Control Insignificant Variations (p-Value higher than 0.2)

Existing Elements Which We Think Worked Well:

Compared To A +13% higher than 0.20).

Differences Which We Think Have Contributed To The Effect:

Compared To A +27% 0.05). Nice!

Differences Which We Think Have Contributed To The Effect:

Compared To A +14% higher than 0.20).

Differences Which We Think Have Contributed To The Effect:

One Less Subheader

Differences Which We Think Have Contributed To The Effect:

One Less Subheader

Compared To A +19% between 0.2 and 0.05).

Differences Which We Think Have Contributed To The Effect:

One Less Subheader

Compared To A +21% between 0.2 and 0.05).

Differences Which We Think Have Contributed To The Effect:

One Less Subheader

Visitor Types Tool Used Page Scope

Test Duration Design & Build Time # of Variations

Primary Metric: Contest Signups Test Type

Variations are grey if

FINDING Natural Language Works

FINDING Input Field Styles Matter

Looking Back & Key Learnings

Having a “Safe Bet” Variation

It’s Good Collect Multiple Metrics

Additional Variation Ideas

H Top Aligned Fields With Explanations

I Fewer Distracting Links (Official Entry Rules)

J Inline Entry Rules

GoodUI DATASTORIES - ISSUE #5 - SEPTEMBER 2014

You might also like