Handbook

The Complete Conversion Rate Optimization course

Ruben de Boer - conversionideas.com

Introduction
 Fix
Prioritize IT backlog

Test Combine
Prioritize Hypotheses Test
insights

Pre-test analysis

Create

Optimization research
Test live
Data & User Expert opinion & best
science research practices
Web analytics User feedback Expert review
analysis Learnings Analysis
Science & Recordings Best practices
psychology De
Usability Competitor analysis pl
testing
oy
?
Customer Psychology basics
service

Reporting IT backlog
 Hierarchy of evidence
low

Meta-analysis

Randomized controlled trial

(A/B test)

n o
isi
ec
Data & science
fd
ko
ris

User research

Expert opinion, best practices &

source: Ton Wesseling high competitors
Deals
Thanks to the growing popularity of my CRO courses, I've secured exclusive deals with several
great vendors. These deals are a great way for you to practice the theories from this course and
an opportunity to get these tools with an extended free trial unavailable elsewhere.

Please note that some of the links to these deals are affiliate links. This means that while you get
exclusive free trials, I may earn a small commission if you decide to purchase through these links
(this will not cost you anything extra). This commission helps support my work and allows me to
continue offering valuable content. However, please rest assured that my primary intention is to
provide you with the best resources and deals to enhance your learning experience. I only
partner with vendors I genuinely believe will benefit you and those I have worked with for many
years.

Keep an eye out, as I will always be on the lookout to negotiate more deals to further assist you in
your CRO journey. You can find the current deals on the next page.

Request a tool: https://airtable.com/appwbuVmNBSdxR5Lw/shr0wetErAUSEBZUO

 Deals

Name Tool Deal URL

Convert A/B testing tool 30 days free trial https://www.convert.com/free-trial/ruben-de-boer/

(No Credit Card required)

Mouseflow Heatmaps, session 28 days free trial https://app.mouseflow.com/sign-up?affiliateToken=7yV

recording, funnel & (automatically applied 2GM3it06F_dHKAfohgA&coupon=conversionIdeas
form analysis, after signing-up with this (No Credit Card required)
URL)
surveys

Hotjar Heatmaps, session 30 days free trial https://invite.hotjar.com/ruben-de-boer

recording, surveys, (No Credit Card required)
feedback button,
user interviews
 Deals

Name Tool Deal URL

Lyssna User research 33 days free trial https://app.lyssna.com/users/sign_up?utm_source

=conversion-ideas&utm_medium=student-access&
utm_campaign=affiliate-partner
(No Credit Card required)

Zuko Form analysis 3 months free trial https://app.zuko.io/signup?utm_source=Udemy

(No Credit Card required)

After sign-up, send an email to support@zuko.io

claiming the extended trial:
“Hi,

I signed up using the link provided by Ruben

(Conversion Ideas) with this email address: [your
email]. Please activate my extended free trial.

Kind regards,
[your name]”
Conversion
optimization research
Conversion Rate Optimization research

To be successful at Conversion Rate Optimization, research is of great importance.

Good and proper research and findings from that research will result in valuable
insights into your website visitors’ needs, motivations, and behaviors. It will also
give better input for your A/B testing plan. It will result in better hypotheses and,
therefore, much more successful A/B tests.
Data
research
Conversion optimization research
GA4 terms

● User: A visitor on your website.

● New user: A visitor on your website for the first time.
● Session: A visit to your website.
● Engaged session: A session with engagement (the session lasted longer than 10
seconds, or had at least one conversion event, or had at least two page views).
● Average engagement time: Average time users are active on your website during
a session.
● Total revenue: Sum of all purchases and subscriptions.
● (Page/screen) view: A view of a page by a user. For apps, this is called screen view.
● DAU / WAU / MAU: Daily/weekly/monthly active users.
● Scroll: User scrolled down to 90% of the page.
Checklist check if everything is tracked

❏ Determine your company, website goals, and micro goals

❏ GA4: Ensure enhanced measurement is enabled
❏ GA4: Exclude internal traffic
❏ GA4: Ensure Google signals is enabled
❏ GA4: Determine your conversion goals
Checklist data analytics

❏ GA4: Check the general data

❏ GA4: Conduct a landing page analysis
❏ GA4: Conduct a funnel analysis
❏ GA4: Analyse your most important segments
❏ Conduct a web interaction research
❏ Check your page speed and mobile-friendliness
❏ Document all your findings
GA4 landing page analysis report
GA4 funnel analysis report
GA4 path exploration report
Resources
Google Merchandise Store GA4 account:
https://support.google.com/analytics/answer/6367342

Google Analytics: https://analytics.google.com

Airtable: https://airtable.com/

Documentation sheet:
https://docs.google.com/spreadsheets/d/17xfdJq62yPNd-ljQ9l8H1l76yMmRnZnMx9dR6_E
Q-m4/edit?usp=sharing

Mouseflow (including extended free trial):

https://app.mouseflow.com/sign-up?affiliateToken=7yV2GM3it06F_dHKAfohgA

Hotjar (including extended free trial): https://invite.hotjar.com/ruben-de-boer

Zuko (send them an email to claim your 3 months free trial after signing up):
https://app.zuko.io/signup?utm_source=Udemy
Resources

Microsoft Clarity: https://clarity.microsoft.com/

Contentsquare: https://contentsquare.com/

Glassbox: https://www.glassbox.com/

Google PageSpeed Insights: https://pagespeed.web.dev/

PageSpeed compare: https://pagespeed.compare/

WebPagetest: http://www.webpagetest.org/

Mobile-Friendly Test: https://search.google.com/test/mobile-friendly/result

Related course

Free course
Conversion Rate Optimization process
& insights in Airtable

Course content:

● Introduction
● Airtable free version
● Airtable paid version

Get the course on Udemy

User research
& science
Conversion optimization research
Checklist user research

❏ Set up polls (on pages you want to optimize)

❏ Set up a survey
❏ Check recordings of pages with a low engagement rate
❏ Conduct a usability test
❏ Conduct a web accessibility analysis
❏ Talk to customer service
❏ Collect user feedback
❏ Optional: other forms of research like a 5-second test or preference test
❏ Read scientific articles on your market, product and target audience
❏ Document all findings
Survey questions

● Describe yourself in a few words (e.g. I'm a Male 26 yr old designer with a passion for cars)
● What are you using [product/ service] for?
● What is the primary benefit that you have received from [product/ service]?
● What made you purchase [product/ service] from us?
● Did you consider any alternatives to our product / buying from us? If so, which ones?
● Which doubts and hesitations did you have before completing the purchase?
● What’s the one thing that nearly stopped you from buying from us?
● Which questions did you have, but couldn’t find answers to on the website?
● What was your biggest challenge, frustration, or problem in finding the right product?
● Have you recommended [product/ service] to anyone? (Please explain how you described it)
● What type of person do you think would benefit most from our [product/ service]?
● How can we improve [product/ service] to better meet your needs?
● If you were in charge of our website for one day, what would you change?
● Did you order before with us?
Possible questions in an exit intend poll

● Is there anything preventing you from finalizing your purchase with us today?
● What frustrates you when searching for the right [product/ service]?
● Do you have any questions you couldn’t find answers to?
● What more did you want to know before purchasing with us today?
● Which missing information would make your choice for this product easier?
● What concerns are keeping you from completing your order?
● If you did not make a purchase today, can you tell us why not?
● What were you hoping to find on this page?
● What other content would you like to see us offer?
Possible questions on entry poll

● What is the purpose of your visit today?

● What do you expect to find on this website/ page?
Possible questions on thank you page
poll

● What convinced you to purchase this [product/ service] from us?

● What’s the one thing that nearly stopped you from completing your order?
● Who will use the product you just bought? Me / Someone else?
● Was there anything about the checkout process we should improve?
● What was your biggest fear or concern about purchasing from us?
● What's the biggest question you had about our pricing?
Possible questions on a general poll

● What do you find important when purchasing [product/ service]?

● What information would you need to feel comfortable purchasing [product/ service] today?
● Does this page contain the information you were looking for?
● On this screen, it seems like I should be able to …
● If you could change anything on this page, what would you have us do?
● What was going on in your day that made you search for a solution?
● When you weren’t using [solution] how did you solve this problem?
● If you could no longer use [solution] what would you miss the most?
● What did, and didn’t you like about other solutions you had tried?
● What’s the number 1 thing you’d mention to a friend if you wanted to convince a friend?
Questions for a 5-second test

● What do you think this page was about?

● What do you think you can do on this website?
● What does the company sell?
● Why should you buy from this company?
● What’s your first impression of the website?
● What grabbed your attention?
● How trustworthy do you find this website?
Resources

Mouseflow (including extended free trial):

https://app.mouseflow.com/sign-up?affiliateToken=7yV2GM3it06F_dHKAfohgA

Hotjar (including extended free trial): https://invite.hotjar.com/ruben-de-boer

Microsoft Clarity: https://clarity.microsoft.com/

Lyssna (former UsabilityHub): https://www.lyssna.com/

Optimal workshop: https://www.optimalworkshop.com/

Google Scholar: https://scholar.google.com

Sci Hub: http://sci-hub.tw/

Scemantic Scholar: https://www.semanticscholar.org

Resources

Web Disability Simulator - Chrome extension:

https://chrome.google.com/webstore/detail/web-disability-simulator/olioanlbgbpmdlgjnnampnnlohigkjl
a

YouTube video on navigating a website without a mouse: https://www.youtube.com/watch?v=dEbl5jvLKGQ

Expert review,
psychology basics
& competitor
analysis
Conversion optimization research
Checklist expert review

❏ Check for clarity issues on the website

❏ Check for relevancy issues on the website
❏ Check for friction issues on the website
❏ Check for focus issues on the website
❏ Check for motivation issues on the website
❏ Strengthen motivation with the reciprocity principle
❏ Strengthen motivation with the commitment & consistency principle
❏ Strengthen motivation with the social proof principle
❏ Strengthen motivation with the authority principle
❏ Strengthen motivation with the liking principle
❏ Strengthen motivation with the scarcity principle
❏ Conduct a competitor analysis
❏ Document all findings
Resources

Google’s UX playbook for retail:

https://services.google.com/fh/files/events/pdf_retail_ux_playbook.pdf

Unbounce (including 20% off on first 3 months):

https://invite.unbounce.com/ruben-de-boer
Related course

Online Psychology and Persuasion for

Conversion Optimization

Course content:

● Introduction to Online Psychology

● Building a behavioral model with online
psychology
● Dual processing theory
● Cognitive Ease
● Motivation & risk
● Attention & perception
● Choices & memory

Get the course on Udemy

Combine insights,
prioritization, and
hypotheses
Combine insights, prioritization, and
hypotheses

We completed our optimization research. The next step is to combine all insights,
create behavioral hypotheses, test ideas and prioritize.
Checklist combine insights

❏ Create behavioral hypotheses

❏ Create test ideas by combining your research insights
❏ Choose a prioritization framework
❏ Prioritize your test ideas based on PIE, ICE, or PXL for your first 20 tests
❏ Create an automated, evidence-based prioritization framework
❏ Create test hypotheses for the test ideas with the highest priority
Resources

PXL framework: https://cxl.com/blog/better-way-prioritize-ab-tests/

Evidence-based prioritization framework:

https://www.linkedin.com/pulse/5-steps-create-evidence-based-automated-mo
del-feedback-ruben-de-boer/

Evidence-based prioritization framework in Airtable:

https://www.linkedin.com/pulse/how-set-up-your-evidence-based-prioritization-
video-ruben-de-boer/
Testing phase
Pre-test
analysis
Testing phase
Pre-test analysis

Before you start testing, you need to conduct a pre-test analysis to determine if
you have sufficient data to run your A/B test and to calculate how long the test
should run for statistically valid results.
GA4 MDE calculation report
Regular expressions
Character What it does Match

. matches any character .ate matches mate, late

matches one or more of the

+ ate+ matches atee, ateee
previous characters

matches zero or more of the

* 100* matches 10, 100, 1000
previous item

| or 1|10 matches 1 and 10

^ begins with ^10 matches 100, 10x

$ ends with 10$ matches 110, x10

the adjacent character \. is a .

\
should be interpreted literally \$ is a $
Checklist pre-test analysis

❏ Select your hypothesis with the highest prioritization score

❏ Determine the KPI (should be part of the test hypothesis)
❏ Calculate MDE, test duration, and minimum sample size
Resources

Speero calculator: https://speero.com/ab-test-calculator

Online Dialogue’s MDE sheet:

https://onlinedialogue.nl/conversie-optimalisatie-tools/sample-size-berekenen/

Regex101: https://regex101.com/

Regex for Google Analytics: https://support.google.com/analytics/answer/1034324

Power analysis in A/B testing:

https://towardsdatascience.com/understanding-power-analysis-in-ab-testing-14808e8
a1554

A/B testing statistics: https://cxl.com/blog/ab-testing-statistics/

Setting up an
A/B test
Testing phase
Setting up an A/B test

It is time to set up your A/B test. Before you set it live, you must ensure your testing
tool is set up correctly. Furthermore, you need to design and developer the test.
Checklist setting up an A/B test

❏ Check the setup of your testing tool (hardcoded in the <head> tag of your
website)
❏ Run an A/A test before your first test
❏ Ensure the design matches the test hypothesis
❏ Set up your A/B test in a testing tool (preferably by using JavaScript and CSS)
❏ Create a copy control
❏ Add the Google Analytics events
❏ Optional: add heatmaps, a poll, and extra tracking for each variation
❏ Check your page targetting
Checklist quality assurance

❏ Conduct a thorough quality assurance in the preview mode

❏ Conduct a thorough quality assurance in a live setting (IP address, query
parameter, or with a unique cookie)
❏ Does the variant look good on all devices, screen sizes, and browsers?
❏ Do all functionalities work?
❏ Is data tracked?
❏ Check your audience targeting
Tracking codes

If any of the following codes do not work, a developer,(technical) web analyst, or

support staff from your testing tool should be able to make it work for your website
with a minor adjustment.
Code to send event to GA4

var waitForGA = setInterval(function () {

if (ga) {
clearInterval(waitForGA);
gtag("event", "ABtest | ID1 | variant");
}
}, 100);
Click event to send event to GA4

$("SELECTOR").click(function(){
gtag("event", "AB test | ID1 | v1 sentence click");
});
Element in view with GA event - jQuery

var scrollEventSent = false;

//element in view
$.fn.isInViewport = function() {
var elementTop = $(this).offset().top;
var elementBottom = elementTop + $(this).outerHeight();
var viewportTop = $(window).scrollTop();
var viewportBottom = viewportTop + $(window).height();
return elementBottom > viewportTop && elementTop < viewportBottom;
};

$(window).on('resize scroll', function() {

if (!scrollEventSent && $('.featuredProdName').isInViewport()) {
//place experiment trigger or GA event here
gtag('event', 'AB test | ID1 | element in view');
scrollEventSent = true;
console.log('element in view');
}
});
Element in view with GA event - JS
var scrollEventSent = false;

window.addEventListener('scroll', function() {
if (!scrollEventSent && isInViewport('.et_pb_row_4')) {
scrollEventSent = true;
console.log('element in view');
//place experiment trigger or GA event here
gtag('event', 'AB test | ID1 | element in view');
}
});

function isInViewport(selector) {
var element = document.querySelector(selector);
var elementTop = element.offsetTop;
var elementBottom = elementTop + element.offsetHeight;
var viewportTop = window.pageYOffset;
var viewportBottom = viewportTop + window.innerHeight;
return elementBottom > viewportTop && elementTop < viewportBottom;
}
Code to place a cookie in your browser

javascript:document.cookie="abtest=id1"
Resources

Convert (including free trial): https://www.convert.com/free-trial/ruben-de-boer/

VWO: https://vwo.com/

Mouseflow heatmap tracking:

https://help.mouseflow.com/en/articles/4312070-custom-variables

Hotjar heatmap tracking : https://help.hotjar.com/hc/en-us/articles/360042781493

Clarity heatmap tracking: https://docs.microsoft.com/en-us/clarity/custom-tags

Event tracking tracker Chrome extension:

https://chrome.google.com/webstore/detail/event-tracking-tracker/npjkfahkbgoagkfpkidpjdem
jjmmbcim

Browserstack: https://www.browserstack.com/
Resources

Edit This Cookie Chrome extension:

https://chrome.google.com/webstore/detail/editthiscookie/fngmhnnpilhplaeedifhccceomclgfbg

Article multiple tests at the same time:

https://blog.statsig.com/embracing-overlapping-a-b-tests-and-the-danger-of-isolating-exper
iments-cb0a69e09d3
Related course

Coding for A/B testing: Run more A/B

tests, find more winners

Course content:

● Introduction
● HTML
● CSS for A/B testing
● Javascript (jQuery) for A/B testing
● Data gathering for A/B testing
● Quality assurance for A/B testing

Get the course on Udemy

A/B test live
Testing phase
A/B test live

You chose your test KPI, you did your pre-test analysis, you set up your test, you
made sure that everything is measured, and you made sure that everything is
working perfectly fine. Now it is time to set the test live, document it, and monitor
your test.
Checklist A/B test live

❏ Set the test live

❏ Inform the stakeholder on the test being live
❏ Check if data is coming in correctly after two hours
❏ Check data every day (including SRM alert)
❏ Optional: create an automated report to keep track of your tests
GA4 A/B test check
Resources

Sample ratio mismatch calculator:

https://docs.google.com/spreadsheets/d/1L45L517kVBD2QATg73bSa-qWRpiYOJNvQZtQjyR8kOc/e
dit#gid=0

Sample ratio mismatch paper:

https://www.researchgate.net/publication/334720020_Diagnosing_Sample_Ratio_Mismatch_in
_Online_Controlled_Experiments_A_Taxonomy_and_Rules_of_Thumb_for_Practitioners

Optimizely variant switcher Chrome extension:

https://chrome.google.com/webstore/detail/conversioncoms-optimizely/ejklmdggogcbjpehmmj
nebehjafnkiaj?hl=en-GB

VWO variant switcher: https://andrescholten.nl/live-visual-website-optimizer-variant-switcher/

Resources

Automated keep track of the A/B test sheet (Universal Analytics):

https://docs.google.com/spreadsheets/d/1ooRvn1n1ri4EBSRQxrIdJmzZDNnX6yUG6YgWAvV7SBQ/ed
it?usp=sharing

Dimension and metrics explorer for GA API (Universal Analytics):

https://ga-dev-tools.appspot.com/dimensions-metrics-explorer/
Analyse the A/B test
Analyse the A/B test

Did your test reach the minimum sample size and test duration? Then it is time to
stop the test and start analyzing the data to gather results and learnings and
define the next steps.
Checklist analyze the test

❏ When you reach the test duration, check if there are enough users in your test
❏ Analyze your test results
❏ Calculate the Bayesian probability
❏ Calculate the business case (in case of a winning test)
❏ Segment your test result (if you have sufficient data)
❏ Optional: Automate your test analysis
Checklist learnings & reporting

❏ What can you learn from your test results?

❏ If it was not a winner: was the hypothesis wrong, or the execution?
❏ What could this result say about your customers?
❏ What did we learn in relation to previously completed experiments?
❏ Does your test confirm a behavioral hypothesis?
❏ What other A/B tests can you come up with based on these results?
❏ Create a nice report
❏ Share it with the stakeholder in your organization
Checklist finalize the test

❏ Archive the test

❏ Add new test ideas to test plan and prioritize them
❏ Prioritize other tests in your test plan, based on finalized test
❏ Document everything
GA4 A/B test analysis report
GA4 A/B test analysis report
GA4 A/B test analysis report
GA4 Average purchase revenue report
GA4 Non-binomial KPI report
Resources

Bayesian vs. Frequentist article: https://cxl.com/blog/bayesian-ab-test-evaluation/

Article on statistical significance:

https://guessthetest.com/finally-statistical-significance-clearly-explained/

Bayesian calculator: https://abtestguide.com/bayesian/

Frequentist calculator: https://abtestguide.com/calc/

Bayesian and frequentist calculator: https://speero.com/ab-test-calculator

Blast Analytics (non-binomial calculator): https://www.blastanalytics.com/rpv-calculator

Resources

Article on statistics non-binomial metrics:

https://blog.analytics-toolkit.com/2017/statistical-significance-non-binomial-metrics-revenue-ti
me-site-pages-session-aov-rpu/

Evidence-based prioritization in Airtable:

https://www.linkedin.com/pulse/how-set-up-your-evidence-based-prioritization-video-ruben-de-boer/

Example report:
https://docs.google.com/presentation/d/1jkprJ2PQQaRsyDIZ8qU4DsBT03Ma3t7HxHxk5mFfio0/edit?usp=shar
ing

Automated analysis Sheet (Universal Analytics):

https://docs.google.com/spreadsheets/d/1_N-JRyEsiSHoUq7Iwa_y-FIG6RR476m-mGR7RnVzUcA/edit?usp=
sharing
Personalization
Personalization

Personalization is the process of tailoring experiences to specific subsets of users

based on information a company has learned about an individual. These
experiences can take place on multiple channels like websites, emails, and online
ads.

Just like A/B testing 10 years ago, personalization is still in the beginning stages.
There is still much to learn and discover. It is, however, becoming bigger and
bigger. Therefore, knowledge about this topic will make you a better optimizer. It
helps you to get started with personalization and talk to vendors.
Checklist personalization

❏ Start with personalization if you have a mature CRO/experimentation process

and team
❏ Start with rule-based experimentation and expand step-by-step
❏ Do your data, user and scientific research
❏ Check your A/B test results to find relevant segments
❏ Generate test ideas and prioritize
❏ Add segments to your prioritization framework
❏ Calculate your sample size and test duration
❏ Design, code, run, and analyze your experiment
❏ When you are successful, add more rules and eventually use machine
learning
How to become even
more successful
How to become even more successful

Your success as a conversion optimization specialist is usually your number of A/B

test winners and the number of learnings you got from those experiments.

Run as many experiments as you can, learn from every experiment, and get people
involved.
Checklist how to become even more
successful

❏ Keep the success formula in mind: Success = chance * frequency

❏ Increase your frequency
❏ Decrease the ideation and design time
❏ Learn to code
❏ Decrease analysis time
❏ Increase your chance
❏ Spend more time on your optimization research
❏ Spend more time on your post-test analysis
❏ Use an evidence-based prioritization framework
❏ Build a knowledge database
Checklist how to become even more
successful

❏ Overcome the confirmation bias

❏ Get people involved
❏ Improve and extend the CRO team
❏ Get the right tools
Resources

Podcast on change management - with me as the guest:

https://www.firstprinciples.ventures/ep007-ruben-deboer-experimentation-change-management

Article - How to motivate your organization to adopt CRO:

https://www.linkedin.com/pulse/how-motivate-your-organization-adopt-cro-ruben-de-boer

Article - Experimentation mindset for an experimentation culture:

https://www.linkedin.com/pulse/experimentation-mindset-culture-ruben-de-boer/
Resources

Article - Know your stakeholders for an experimentation culture in 5 steps:

https://www.linkedin.com/pulse/know-your-stakeholders-experimentation-culture-5-steps-ruben-de-bo
er/

Article - Get test ideas and boost the experimentation culture:

https://www.linkedin.com/pulse/get-test-ideas-boost-your-experimentation-culture-ruben-de-boer/

Article - The blocking influence of CRO specialists on organizational change:

https://www.linkedin.com/pulse/blocking-influence-cro-specialists-organizational-change-de-boer/
Related course

Change Management: from CRO to

Experimentation Maturity

Course content:

● Introduction
● Assessing the current situation
● Your change management plan
● Implementing the change
● Evaluate, reflect & next steps

Get the course on Udemy

Questions?
ruben@conversionideas.com
LinkedIn
Website
Learn more?
Change Management: from CRO to
Experimentation Maturity

Course content:

● Introduction
● Assessing the current situation
● Your change management plan
● Implementing the change
● Evaluate, reflect & next steps

Get the course on Udemy

Conversion Rate Optimization process &
insights in Airtable

Free course

Course content:

● Introduction
● Airtable free version
● Airtable paid version

Get the course on Udemy

Online Psychology and Persuasion for
Conversion Optimization

Course content:

● Introduction to Online Psychology

● Building a behavioral model with online
psychology
● Dual processing theory
● Cognitive Ease
● Motivation & risk
● Attention & perception
● Choices & memory

Get the course on Udemy

Coding for A/B testing: Run more A/B
tests, find more winners

Course content:

● Introduction
● HTML
● CSS for A/B testing
● Javascript (jQuery) for A/B testing
● Data gathering for A/B testing
● Quality assurance for A/B testing

Get the course on Udemy