THE ADVANCED GUIDE TO

Scaling a Conversion
Optimization Program

1
 Introduction 04

Table of Chapter 1 05

Contents
Creating and promoting a culture of experimentation

Chapter 2 08
Generating high-impact, data-driven hypotheses repeatedly

Chapter 3 15
Building a long-term experimentation roadmap

Chapter 4 21
Setting program goals and measuring progress

Chapter 5 25
Ensuring continuous learning and inculcating improvements

Chapter 6 29
Growing a winning optimization team

Wrapping Up 32

About VWO and HubSpot 32

2
 “Our success at Amazon
is a function of how many
experiments we do per year,
per month, per week, per day.”

Jeff Bezos
CEO, Amazon

3
Introduction
It’s 2019, and conversion optimization is no longer
a buzzword. If you’re a marketer, product manager,
website owner, or in anyway responsible for
improving conversions on a digital property,
it’s a lever you can’t do without.
A successful conversion optimization program
requires a strategic, methodical approach to
develop it. In this guide, we will focus on how you
can work towards scaling your existing conversion
optimization program to ensure the same.
If you’d like to take a step back and get started from
scratch, we’ve got you covered. In the previous edition
of this eBook, VWO and HubSpot got together to create
The Complete DIY Guide To Improving Conversions In
60 Days, a time-bound resource which aims to enable
budding optimizers build an end-to-end conversion
optimization program.
Let’s get started!

This eBook is divided into 6 chapters. Every chapter

will walk you through one indispensable element of
scaling a conversion optimization program.

4
 Why adopt a culture
of experimentation?
How is it that some companies can consistently move
faster than others and deliver delightful experiences?
CHAPTER 1 Why are some organizations able to innovate more
quickly to deliver those experiences all along the

Creating and customer’s journey?

promoting What sets today’s most innovative companies apart

from their peers isn’t some kind of magic spell;

a culture of
instead, these companies have invested in creating
and promoting a culture of experimentation, and

experimentation
made optimization at scale a core business practice.

Having adopted this approach has helped businesses

like Amazon, Netflix and Google to empower different
teams and functions to collaborate on improving the
customer experience.

A culture of experimentation is geared toward

incremental, consistent improvements coordinated
across all business functions to meet executive goals.

5
How to build a culture
of experimentation?
Rome wasn’t built in a day, the same is true for the
effort required in fostering a culture and mindset
of experimentation within a company. Here’s how
you get started:
Get stakeholders buy-in,
establish basic principles
Like a waterfall, culture too flows from the top
to the bottom. To inculcate a healthy culture of
experimenting and optimizing, it is imperative
that the top management complies.
If the top management is on the same page, it’ll
realize the need for optimization and allocate
sufficient budget for it. This will make it easier to
execute any optimization program.
The idea of optimization can be sold to the top
management by highlighting its key benefits.
You can show how the returns on efforts out-
weigh the investment, and strengthen your pitch.
Get the entire team involved
to build a mindset
While the top management provides resources for
an optimization program, the success of the program
also depends on the understanding and acceptance
of the approach by the entire team.
Get everyone involved and teach them enough
so they can contribute. You can hold information
sessions or talks to review the basics of optimization.
“If you want to incorporate a culture
of optimization, you need to allow and
encourage anyone to challenge or at least
question the data or ask how you could test
an idea. But when someone says or asks
something, be sure to ask them to back up
their claim with data. They need to be able
to validate any of their claims or thoughts,
or at least give reasons for them.”
Michael Heiligenstein,
Director of SEO at Fit Small Business
6
Become data-driven first,
intuition-driven never

Before you begin optimizing your business

processes, it is imperative to take stock of the
current standing. You need to assess how you
1.8
trilion
are placed currently and set benchmarks for
improvements accordingly.

A good optimization program would be

rooted in in-depth data and research of what
we are optimizing and for which end goals.
When your employees start questioning
subjectivity and validate their decisions with
both data and insight, it ultimately makes
your organization a more efficient one.
2018 2018 2020 2021 2018

According to Forrester Research,

Insights-driven businesses are While these three points outline the essence of how you can
growing by more than 30% build a promote a culture of experimentation and optimization,
annually and are on track to you need to come up with ways unique to your own organization.
earn $1.8 trillion by 2021. For inspiration on the same, this post is a good place to start.

7
 CHAPTER 2

Generating high-impact,
data-driven hypotheses
repeatedly

8
Coming up with an idea that What winning companies optimize for
you want to test isn’t tough.
Coming up with one that you
should test can be. Acquisition Metrics Behavior Metrics Outcome Metrics

Source
More often than not, optimizers base their
testing on intuitions and best practices,
What siloed teams actually end up optimizing
eventually yielding unfavorable results.
for because of bad org/incentive structures

Some others tend to adopt a myopic approach by keeping

only a single aspect of the marketing funnel (acquisition/ For testing to make any sense
behavior/outcome) in mind, without sight on the long-term (and therefore for the result to
goals. In order to scale your conversion optimization program, have any value), you need to first
it is important to structure it as a methodical process. learn how to continually generate
high-impact, data-driven
The process involves conducting thorough research, asking hypotheses repeatedly.
the right questions, digging for answers in the problem areas,
running smart tests, and eventually deriving valuable results.
According to an Econsultancy report on CRO, companies with
a structured approach to improving conversions were twice as
likely to see a large increase in bottom line.

9
Formulating Observing Data

hypotheses using You may have all the data lurking in your records, but all that data has to
be distilled into a logical hypothesis. This is explained by the graph below.
observations
Instead of randomly testing ideas that
you ‘feel’ are good, the focus should Wisdom
be on building a solid hypothesis Ability to use the knowledge
that maximizes chances for winning. to understand the reasons
Knowledge in larger contexts
Theoretically, there are two approaches
to building a hypothesis. Contextualized
information
Information
inductive approach — i.e. begin with Data with analyzed relationships
brainstorming a set of ideas and then and connections

look at the data to validate those ideas Data

and form a hypothesis. Simple and objective
Understanding

deductive approach of looking at
patterns in your observations first, and
then deducing a hypothesis for testing.
Either way, the most crucial part in forming
a strong hypothesis is the research that
goes behind it.
Source
Evidently, putting data into context with
a certain level of understanding is the key.
The “simple objective facts” or ideas could be transformed
into a well-structured hypothesis by understanding your
website analytics and aligning it with your business objectives.

10
 Analyzing your Website Analytics
Unclear UI
Your analytics data is your first port of
No one Low Contrast
sees them call while formulating a hypothesis. With
the wealth of data that gets tracked, you
Never Announced
Why... could get answers to the most obvious
questions related to the current situation
Can’t show to boss
of your website.

People don’t Too much work involved Website analytics tools like Google
see the value
Analytics and Kissmetrics can show
Easier to do in Excel
No one Why... you quantitative data on how visitors
uses
Reports navigate your website on a site
architecture level. For any observation
No Data Visuals
Why... that you come across from analyzing
The UI is too Don’t understand “Merge” data, ask yourself enough number
confusing
of “why’s” to form a solid hypothesis.
Too many Dropdown
Why...

Needs more testing

Too many Bad Browser support

bugs in them

Response Time Poor

Why...

Source

11

Observing Behaviors
Now that you’ve gained an under-
standing of what visitors are doing on
your website, you would next need to
know why they’re doing it. Below are
the two practices that could help you
identify and eliminate the problem:
Heuristic Analysis
This is when usability experts review
your website to identify any common
usability/design issues. Each review
is based on a set of usability best
practice principles and/or design
consistencies. One of the most
popular heuristic analysis frameworks
is defined by Jakob Nielsen.
Ten Usability Heuristics
by Jakob Nielsen
Visibility of system status
Give the users appropriate
feedback about what is going on.
User control and freedom
Support undo, redo and
exit points to help users
leave an unwanted state
caused by mistakes.
Aesthetic and minimalist design
Don’t show irrelevant or rarely needed
information since every extra element
diminishes the relevance of the others.
Flexibility and
efficiency of use
Make the system efficient
for different experience
levels through shortcuts,
advanced tools and
frequent actions.
Help and documentation
Make necessary help and
documentation easy to find
and search, focused.
Match between system
and the real world
Use the real-world words,
concepts and conventions
familiar to the users in a
natural and logical order.
Error prevention
Prevent problems from occurring:
eliminate error-prone conditions
or check for them before users
commit to the action.
Consistency
and standards
Follow platform conventions
through consistent words,
situations and actions.
Recognition rather than recall
Make objects, actions, and options visible
at the appropriate time to minimize users’
memory load and facilitate decisions.
Help users recognize, diagnose,
and recover from errors
Express error messages in plain language
(no codes) to indicate the problem and
suggest solutions.
12
Visitor Behavior Analysis

Next, examining the behavior of current visitors

could help you identify the specific details of the
most pressing issue with your conversion process.

Visitor Behavior Analytics tools like Heatmaps,

Clickmaps, Mouse recording, etc. could tell which
part of the page in specific they spent most time
on (or ignored completely).

Here’s an example:
quantitative analytics can tell you that visitors
are exiting your product page at an alarmingly
high rate. Visitor recordings go a step further to
actually show recorded sessions of your visitors
on the product page, helping you visualize where
the users spend most of the time on the page,
areas they get stuck on, information they don’t

seem to locate,and so on.

13
 Observing Opinions
Your analysis (web analysis or visitor behavior analysis) could
run the risk of narrative fallacy or confirmation bias while
forming a formalized hypothesis. That is where collecting
real-time quantitative and qualitative data via customer
surveys could help. Surveys primarily come in two forms:
On-site Surveys
On-site surveys enable you to receive feedback from your
users via a popup or a layer that the visitor is prompted to
fill up. It is a great mechanism to find out more about your
actual users and validate your hypothesis. You can gather
information about the interests, attitudes or preferences,
straight from the horse’s mouth.
In general, there could be three things that you want to
think about when planning to use on-site surveys:
Why ask the question: Clearly outline the end goal
you’re conducting the survey for. For instance, whether
you want to feedback on website design/content/
relevance, etc.
When to ask the question: Asking the right question
at the right time is important. You could look at your
average time on site and/or page view metrics and ask
questions to visitors who have engaged enough with
your website/page (for qualification reasons).
Which questions to ask: Which questions to ask
depends heavily on your end goals. If you’re doing
a voice-of-customer research to gain insights on copy/
design, open-ended questions are gold. If you’re trying
to quantify customer experience, measuring Net
Promoter Score (NPS) could do the trick.
Here are some additional tips that would make your
website surveys shine, and a guide to surveys to
get you going.
14
 Building a strong hypothesis
is crucial for testing—no doubt.
CHAPTER 3
But what should you do when you have a backlog

Building a of potential hypotheses to test? How should you go

about deciding which one to test first?

long-term Often, organizations do not have a structured

experimentation
approach to conversion optimization and arbitrarily
pick out a hypothesis from the pool of options they
have.

roadmap However, organizations following a structured

approach to it realize the need for a robust
prioritization framework. Let’s first look at why
scaling your optimization program needs a
prioritization framework in the first place.

15
What’s the importance
of a prioritization framework?
Analogous to a calendar, an experimentation roadmap provides
a clear direction to your optimization program and prevents

1
you from doing aimless testing. This framework enables you to
maintain a dedicated schedule for all your CRO activities.

Michal Parizek, Senior eCommerce & Optimization Specialist 2

at Avast, in his interview with VWO, points out the importance
of keeping a testing calendar:
3

Prioritization helps you answer,

“A test calendar helps to keep focus on
important tests being launched on time. “here’s what we’ll
It is also vital for resource planning and for test, in this order,
bringing all stakeholders in a loop. We
and here’s why.”
usually do a quarterly overview of what
tests we’d like to run and then we specify
and add details on a monthly basis.”

16
Setting up your
prioritization framework
You would find a plethora of frameworks for prioritizing
your testing hypothesis. Some of these are listed below:

Widderfunnel’s PIE framework for Prioritization:

A prioritization model by Chris Goward that fosters scoring
(on a scale of 10) on three factors: Potential, Importance,
and Ease.

Bryan Eisenberg’s T.I.R Model: Prioritization calculated Karl E. Wiegers, in his paper on Prioritizing
by multiplying the individual scores of Time, Impact, and Requirements states this simple rule for prioritization:
Resources.

Monetate Model: Prioritization based on the level of difficulty

“When setting priorities, you need
of the test, creative effort required, and potential of the test.
to balance the business benefit that
Hotwire’s Points Model: Binary scoring on various questions
each function provides against its
such as those based on reach, lift, strategic fit, and design. cost and any implications it has for
the product’s architectural founda-
So which of these should you decide to apply to your tion future evolution.”
conversion optimization plan? On what factors would
you base your prioritization framework?

17
Determining the
Potential Impact
Business Objectives
Depending on the research you have put in
to build your hypotheses, you can estimate
the impact that it might yield. The more
insight-driven your hypothesis is, the more Website goals
confidence you can have on its success.
For example, your heatmap analysis could
indicate that a large chunk of your visitors
are not able to locate your CTA, making you Key Performance Indicators
confident that changes made to it will yield
positive results. Some of the other factors to
estimate the impact of the test are:
Target metrics
Alignment with Business Objectives

Prioritization eventually becomes a matter of answering these questions:

For each hypothesis you want to test, there
would be a target metric that you will track.
It is imperative that this key metric is in
Which business goals are Which features or pages
alignment with your business objectives.
we trying to improve at are closely associated with
ConversionXL explains the linear correlation
this moment? the business goal?
between the business objectives and
target metrics here:

18
Location of the Test

The page on which the test is performed is also critical to understand its potential. You can look for:

Most critical pages
In terms of the business goals they
are seeking to fulfill. For instance,
a pricing page would be more
critical to a SaaS business than say
the “About” page.
Most visited pages
Pages with high traffic volume are more
likely to yield quicker results. The graph
below indicates how higher the traffic,
the lesser time it takes for the test to
reach a conclusive result.
Pages with expensive visits
When choosing between two pages
with similar traffic and conversion rates,
pick the one with a higher cost of
traffic (through paid advertisements)
for a better split-testing ROI.

450

400

350

300

250

200

150

100

50

0
500 1000 1500 2000 2500 3000 4000 5000 6000 10000 20000

Source

19
Determining the Potential Ease

To estimate the effort required to test your hypothesis, consider the following factors:

Availability of Resources Cost of the Test Duration of the Test

A full-fledged CRO program would
require help from a designer,
copywriter, developer and so on.
By looking at the bandwidth and
strengths of your team/members,
you can estimate which particular
test should be conducted first.
The ease with which the test can
be conducted is also dependent
on the cost of conducting the test.
The cost would typically include the
cost of resources and tools used in
the optimization process.
To conclude, the potential ease
of conducting the test can also
be estimated by the time it would
take to finish the test. You can
calculate for how long should you
conduct a test by using VWO test
duration calculator.

A cumulative assessment of the two factors shared could give you a simple
framework for prioritizing. For instance, you would prioritize tests with high
impact and low effort more than the ones with say, low impact and high effort.

20
 CHAPTER 4

Setting program
goals and measuring
progress

21
Successful conversion optimization programs and teams set goals for experimentation and then measure their
performance for those goals over time. To perform well, it’s important to track both the volume and velocity of
experiments you are running to understand if you are improving over time.

Measuring the time it takes an experiment to go from an idea to deployment, and how many experiments that you have
at each stage of the process - from ideation and development, to testing and analysis - can also yield valuable insights.

Step-by-step STEP 1

breakdown of Write down your business goals

goal setting
Here are the four steps we feel
essential when you’re trying to
set goals and measure the
progress for your conversion
optimization program.
Conversion Optimization should be based
around business goals. It is like building a house.
You can’t paint the walls until you have walls. You
can’t put up walls until you lay the cornerstone. If
you try to go out of order you’ll end up with one
that couldn’t stand up to the stormy days.
So start by looking at your overarching business
goals. What do you want to achieve in the next
one year? Two years? Five years?

22
STEP 2
Cascade business goals into
conversion goals
Once you have established a solid foundation,
you can start building the walls. Conversion rate
optimization is all about your digital properties.
So the most important question to ask yourself
now is: how can my website or mobile app
contribute to my business goals? Then, you can
set specific goals that will have a measurable
impact on our business goals.
STEP 3
Assign metrics for
each conversion goal
Now it is time to make your goals measurable
(and potentially break them down further).
For each of your website goals, you need to
have a metric or set of metrics to measure.
These metrics will then be the measuring
stick by which you choose to optimize and
determine the success of our individual
conversion optimization campaigns.
STEP 4
Optimize to improve those
metrics over time
With all the hard work you did in the previous steps
will pay off now. Now that the foundation has been
laid, you can start building on it.
Each of your optimization tests should be directly
relatable up the chain and back to your business
goals so that you can track measurable success. Again,
don’t create tests just for the sake of creating them.
Make sure they have a purpose. Selecting and creat-
ing conversion optimization tests becomes incredibly
simple and straightforward as long as you’ve got your
foundation laid in advance. Conversion optimization is
only useful if it is in service of a business goal, so make
sure you aren’t getting ahead of yourself.
23
Macro conversions and micro conversions
Here’s another important thing to consider about your
conversion optimization program: not all visitors to your Macro conversions, on the other hand, are
site are at the same buying decision stage. Some are just actual sales conversions, like a checkout for an
looking to learn things, some are actively prospecting, and eCommerce business or making a call to your
some others are ready to buy today. The fact that visitors Sales team.
will be at all different stages is why you need to track and
optimize for conversions at all stages for every user.

Are you looking for more free trial sign-ups, more Micro conversions are low-involvement
average revenue per visitor, or top-of-the-funnel commitments from a visitor, like an ebook
ebook downloads? download or a newsletter subscription.

Enter macro conversions and micro conversions.

It may seem like macro conversions are all that matter.
Micro Macro However, micro conversions are the baby steps that
visitors start with to get to know your brand, eventually
leading them to become customers. Good conversion
optimization requires tracking both.

24
 CHAPTER 5

Ensuring continuous
learning and inculcating
improvements

25
Developing sound data-driven hypotheses and conducting experiments can uncover surprising findings filled with
useful insights. The potential business impact of these insights hinges onYour team’s ability to identify and interpret
them. Whether your results are good, bad, or inconclusive, they are still results, and there is always something to learn.

Investigating and For example, adding a new feature into your website
experience may have a much greater on mobile visitors
analyzing your outcomes than desktop visitors. Asking some of the following
questions can help you analyze your results across
Before you can take action you’ll need to evaluate the experiments more effectively:
results of your experiment to determine whether tests
were winning, losing, or inconclusive. During the Do any specific segments behave
investigation phase, start by comparing results to your differently from the overall audience?
original business goals we discussed in the previous
chapter, and your hypotheses to bring the tests full circle. If so, what are the attributes and
goals of this particular segment?
Refer to your campaigns’ overall results to get an idea of
the average visitor’s or user’s behavior, then segment your Why do you think they responded
results to focus on specific segments. This deep dive into to the test the way they did?
audience segments can help you gain actionable insights
and even uncover surprising conclusions. Which variation did your most
valuable visitors choose?

Use the answers to these questions to make data-

driven business decisions based on your results.

26
What to do with your test results
When a Winning Variation is Found
Great! What you should do now is answer
these questions, and proceed as follows:
What is the cost of deploying the change
(engineering hours, design hours)?
Does the expected increase in the
revenue justify the cost involved?
When a Losing Variation is Found
When the variation loses, make sure you:
Look at the research; ensure
that the hypothesis isn’t faulty.
Analyze the test data; do
segmentation to reveal further
insights.
Go through relevant case studies!
It could reveal new perspectives.
Reconstruct the hypothesis to
accommodate new insights that
were missed in the initial research.

Validate research data with Go back to following your

surveys and visual analytics. conversion optimization process.

YES Talk to your engineering and design

NO Hold on to deployment.
teams to get the change implemented.
Use post-test segmentation: can the
Analyze the test data to see if there hypothesis be refined for more impact?
are further opportunities to optimize.
Reconstruct the hypothesis
Use these learning outcomes to fuel
Use these learnings to fuel further
further optimization efforts.
optimization efforts

27
Sharing and evangelizing learnings

Once you have the data with you, promoting company-wide

transparency is just as critical to a successful experimentation
program. It widens the impact of your testing, promotes the
culture you’ve worked so hard to develop, and evangelizes
the importance of continuous iteration.

Start by sharing results with your testing team, relevant

stakeholders, and anyone involved in the ideation process.
Make sure you include the purpose of each test, clearly define
the details and hypothesis, provide details about the impact on
revenue, and share lessons learned – regardless of the outcome.

Don’t be afraid to get creative with your sharing routine.

Distributing something as simple as a “Which Variation Won”
poll that highlights the hypotheses and key learnings can
Don’t be afraid to
increase engagement and get more employees invested
in your experimentation program. get creative with
your sharing routine.

28
 CHAPTER 6

Growing a winning
optimization team

29
The importance of building strong, efficient teams is not a new concept. Teamwork about creating a clear vision,
aligning on objectives, and working as a unit to achieve success. As companies seek to grow and optimize their
experimentation efforts building teams dedicated to testing ideas and concepts is the key to unlocking the path
to relentless innovation. So, what do experimentation teams look like at the enterprise level?

What does an ideal A Copywriter creates copy that will reduce customers’
anxieties, ease friction, and persuade visitors to take the
team look like? desired action.

Since Conversion Optimization is cross-functional, A Web Developer is in charge of developing variations

it involves a team of professionals from different of webpages for A/B tests using optimized code.
departments and functions to coordinate effectively
and make the most out of available resources.
Going forth, you should focus on determining who will
take care of which responsibilities for your optimization
At any given time, an optimization team needs to
needs. Based on the bandwidth and budget at your
collectively possess talent with all of the following skills:
disposal, you can either onboard key hires for these
roles or identify talent from within your organisation
A Conversion Rate Optimization Manager is in charge
and rope them in.
of developing an end-to-end plan for your optimization
program—enumerating and prioritizing essential activities.
An ideal case would be a dedicated professional
possessing one of each skill. However, not every
A Data Analyst monitors your website data and user
organization has the resources to dedicate five
behavior using relevant tools to uncover actionable insights.
employees for it. For many small businesses, a single
team member can also take care of multiple functions.
A Web Graphic Designer will help you design
web pages with a focus on increasing conversions.
30
Invest in ongoing
education and growth
opportunities
The best organizations, that run very mature
experimentation programs, tend to invest heavily in
employee development. That means granting generous
education stipends for conferences, books, courses,
and internal trainings.

Different companies can have different protocols as

well. Airbnb, for example, sends everyone through data Intermediate Google Analytics
school when they start at the company. HubSpot gives
a generous education allowance.
A/B testing mastery course

There are tons of great Conversion

Creating A Conversion Optimization
Optimization specific courses out there
Strategy: A HubSpot Academy Course
nowadays. Some programs you should make
everyone in your team run through are:

CRO Certification Program

31
Wrapping Up
As this guide comes to an end, it would
be beneficial for us to circle back and
Learn more about conversion
reflect on the most important aspect
when it comes to scaling anything in a optimization in HubSpot Academy.
business: culture and mindset. Make sure
you’re able to follow the basic tenets
LEARN MORE
of the same shared in this eBook.

Before you leave, be sure to pay this

guide forward. Share it with your friends,
colleagues, or anyone who you might
think could gain value from it. What’s
more, bookmark it for your own reference,
and be sure to download a copy.

Good luck, and happy optimizing! If you’re looking to explore a platform

for scaling your conversion optimization
program, try out VWO free for 30-days.

SIGN UP FOR FREE TRIAL

32