a/B TesTing for markeTing opTimizaTion

Using Split Tests to Optimize Landing Pages, Email and Calls-to-Action

An Introduction to Using

A publication of

2

An introduction to A/b testing

An introduction to A/b testing

3

is THis Book rigHT for me?
Not quite sure if this ebook is right for you? See the below description to determine if your level matches the content you are about to read.

inTroDUCTorY

This ebook!

Introductory content is for marketers who are new to the subject. This content typically includes step-by-step instructions on how to get started with this aspect of inbound marketing and learn its fundamentals. After reading it, you will be able to execute basic marketing tactics related to the topic.

HUbSPOT’S ALL-IN-ONE mArkETINg SOfTwArE
... brings your whole marketing world together in one, powerful, integrated system.

LEAd gENErATION

U q

bLOggINg & SOCIAL mEdIA

inTermeDiaTe
Intermediate content is for marketers who are familiar with the subject but have only basic experience in executing strategies and tactics on the topic. This content typically covers the fundamentals and moves on to reveal more complex functions and examples. After reading it, you will feel comfortable leading projects with this aspect of inbound marketing.

get found: Help prospects find you online Convert: Nurture your leads and drive conversions analyze: measure and improve your marketing plus more apps and integrations

EmAIL & AUTOmATION

M s

SEArCH OPTImIzATION

request A demo

Video Overview

aDVanCeD
Advanced content is for marketers who are, or want to be, experts on the subject. In it, we walk you through advanced features of this aspect of inbound marketing and help you develop complete Share This Ebook! mastery of the subject. After reading it, you will feel ready not only to execute strategies and tactics, but also to teach others how to be successful.
www.HUbSPOT.COm

LEAd mANAgEmENT

g Y

mArkETINg ANALyTICS

Share This Ebook!

www.HUbSPOT.COm

HUbSPOT. including ebooks and webinars.COm . She has previously helped with HubSpot’s email marketing program and the company’s landing page creation and optimization. Petersburg Times.com and The St.4 An introduction to A/b testing An introduction to A/b testing 5 an inTroDUCTion To Using a/B TesTing for markeTing opTimizaTion Edited by magdalena georgieva maggie georgieva is an inbound marketing manager at HubSpot responsible for creating new offers.COm www.HUbSPOT. ConTenTs 10 gUiDelines for effeCTiVe a/B TesTing wHaT VariaBles YoU sHoUlD TesT /17 a/B TesTing Case sTUDies /31 /39 /9 rUnning & eValUaTing YoUr TesTs ConClUsion & aDDiTional resoUrCes /50 follow me on TwiTTer @mgieVa Share This Ebook! Share This Ebook! www. maggie is a prolific marketing blogger and has also written for publications like bostInnovation.

In fact. In fact. mArkETINgSHErPA LANdINg PAgE OPTImIzATION bENCHmArk SUrVEy. The insights stemming from split tests can drastically improve the conversion rates of your landing pages and the clickthrough rates of your website calls-to-action and email campaigns. the element that is being testing is called “control” and the element that is argued to give a better result is called “treatment. also known as split testing. Just a few small tweaks to a landing page. A/b testing of landing pages can generate up to 30-40% more leads for b2b sites and 20-25% more leads for eCommerce sites. A/b testing costs next to nothing to implement. In this context. is a method of testing through which marketing variables are compared to each other to identify the one that brings a better response rate. Share This Ebook! www. you can easily create more opportunities for your sales team and get ahead of your competitors.HUbSPOT. And although sometimes it’s the more complex elements that need to be revised.COm .” Running A/B tests in your marketing initiatives is a great way to learn how to drive more traffic to your website and generate more leads from the visits you’re getting. you’re missing out on some serious lead generation boost. In other words. Share This Ebook! “In our ROI-driven world of digital marketing. 2011 60% 40% A/b testing. with such an effective lead generation tool. ONLy 40% Of mArkETErS VALIdATE TEST rESULTS. with the right software.HUbSPOT. chances are your competitors aren’t doing A/b testing right.6 An introduction to A/b testing An introduction to A/b testing 7 “ Why should I conduct A/B testing? ” ? Such increases in lead generation can provide a huge competitive advantage for your company.COm www. testing without calculating statistical significance of the data is not much different than not testing at all. email or call-to-action can significantly affect the number of leads your company attracts. some of the most dramatic results come from quick and simple changes.” write marketingSherpa in its report. if you’re not A/b testing.

Let’s dive into A/b testing and start with a look at how to create an effective A/B test and how to measure its significance. multivariate testing combines a series of simultaneous A/b tests. A/b testing allows you to only test one variable at a time.8 An introduction to A/b testing An introduction to A/b testing 9 A/b TESTINg Or mULTIVArIATE TESTINg: wHAT’S THE dIffErENCE? Although these two terms are both often thrown around in the marketing world. multivariate and A/b testing are not the same thing. you might test red vs. if you are testing the background color of your landing pages to determine which one helps you generate the most conversions.COm www. It’s used to pick the better out of two variables. you can also test on the page level. for instance. but many of the practices mentioned in it can be applied to multivariate testing as well.) This ebook covers A/b testing. which means that you are looking at the entire page as the variable. In order to conduct statistically significant multivariate testing. Theoretically.HUbSPOT. multivariate testing. enables you to test many variables simultaneously.HUbSPOT.COm . you need to have a lot of traffic (the type of traffic youTube and google get. CHapTer 1 a/B TesTing 10 gUiDelines for effeCTiVe a/B TesTing mUlTiVariaTe TesTing Share This Ebook! Share This Ebook! www. on the other hand. blue.

say. Remember. Note that by taking the entire page. In fact. TesT one VariaBle aT a Time There are certain guidelines you should keep in mind before you implement your tests. calls-to-action and email. Share This Ebook! Share This Ebook! www. Only test one element at a time.10 An introduction to A/b testing An introduction to A/b testing 11 “ fast. while creating your tests. remember that even a simple change.COm . the small details are often just as important. How would you know which A/b testing can quickly give rise to a number of questions. you can achieve drastic improvement. your results can get muddled pretty easily. don’t conduct multiple tests at the same time. ” #1 change caused the increase in leads? onlY ConDUCT one TesT aT a Time If you test an email campaign that directs to a landing page at the same time that you’re A/b testing that landing page. switching the color of your call-to-action button. call-to-action or email campaign is. and especially if you’re relatively new to the A/b testing method. #2 In order to evaluate how effective an element on your page. you have to isolate that variable in your A/b test.HUbSPOT. Too Although it’s reasonable to think that big. a simple change can drive big results.COm www. sweeping changes can increase your lead generation numbers. can drive big improvements. email or CTA as the variable. Such knowledge will guide you in figuring out how to best optimize your landing pages. what should you start testing? How long should the test last? Things get confusing As a general rule.HUbSPOT. In this section we will cover some best practices that will make it much easier to measure your results and find out which variation performed better. these sorts of changes are usually easier to measure than the bigger ones. #3 TesT minor CHanges.

COm www. so consider starting with it before you continue your optimization with smaller tweaks.HUbSPOT. Instead of testing single design elements. your A/b test might have a positive impact on your landing page conversion rate.COm . call-to-action or email a variable. CliCkTHroUgH raTe mEASUrE AS fAr dOwN THE fUNNEL AS POSSIbLE #5 measUre as far Down THe fUnnel as possiBle ConVersion raTe Sure. leads.HUbSPOT.12 An introduction to A/b testing An introduction to A/b testing 13 YoU Can a/B TesT THe enTire elemenT #4 #5 while you can certainly test a button color or a background shade. design two completely different pages and test them against each other. Now you’re working on a higher level. consider how it affects metrics such as clickthrough rates. You may even see that a landing page that converted fewer prospects produced more sales. This type of testing yields the biggest improvements. but how about your sales numbers? A/B testing can have a significant effect on your bottom line. traffic-to-lead conversion rates. you should also consider making your entire landing page. and demo requests. As you create your A/b test. such as headlines and images. leaDs Demo reqUesTs sales Share This Ebook! Share This Ebook! www.

#7 DeCiDe wHaT YoU wanT To TesT In any experiment. In fact. and the length of time a particular name has been on a list are all factors that may cause large differences in response rates. for example. Share This Ebook! Share This Ebook! www. you don’t have to limit yourself to testing only color background or text size. wording. your test results will not be conclusive or you may draw the wrong conclusions if you do not split your lists randomly. some of the areas you can test might not be instantly recognizable. you can test different target audiences. you need to keep a version of the original element you’re testing. list type. List sources. etc. in email A/b testing each of your email variations must have as similar a group of recipients as possible. original version) • Variation B: Treatment (the optimized version which you expect to perform better) The screenshot below is an example of HubSpot’s advanced landing pages A/b testing tool. As you optimize your landing pages. if you are wondering whether including a testimonial on a landing page would make a difference. Look at the various elements on your marketing resources and their possible alternatives for design. calls-to-action and email. for instance. set up your control page with no testimonials.COm www.HUbSPOT. for instance. spliT YoUr sample groUp ranDomlY #8 In order to achieve conclusive results. from there. set up your unaltered version as your “control:” the landing page. alignment between an email and a landing page. timing. build variations.HUbSPOT. you need to test with two or more audiences that are equal. • Variation A: Control (the unaltered. or “treatments:” pages. callsto-action or email you’ll test against your control.14 An introduction to A/b testing An introduction to A/b testing 15 seT Up ConTrol & TreaTmenT #6 layout.COm . there are a number of variables you can test. when conducting A/b tests. call-toaction or email you would normally use. keep all other aspects of the design and timing identical so you get clean results based on list and nothing else. If you want to compare the performance of two or more lists. you can easily clone the control and modify it to create a treatment page. Then create your variation(s).

or month of the year. Share This Ebook! Share This Ebook! www.16 An introduction to A/b testing An introduction to A/b testing 17 #9 different month. think about how significant your results should be in order for you to decide that the change should be made to your website or email campaign. Set the statistical significance goal for your winning variation before you start testing. If you were to run test A during one month and test b a month later. without simultaneous testing. you may be left second-guessing your results.HUbSPOT.COm www.HUbSPOT. wHaT VariaBles YoU sHoUlD TesT DeCiDe on neCessarY signifiCanCe Before TesTing #10 Before you launch your test. TesT aT THe same Time CHapTer 2 Timing plays a significant role in your marketing campaign’s results – be it time of day. you wouldn’t know whether the changed response rate was a result of the different template or the A/b testing requires you to run the two or more variations at the same time. day of the week. 97-99% statistical significance is usually a good percentage to aim for.COm .

HUbSPOT.HUbSPOT. Share This Ebook! Share This Ebook! www. Should you change the background color of the landing page? How can you modify the language on the call-to-action to attract the most clicks? what if you removed all images from your email campaign? Not all variables are created equal. That has led us to maintain a balanced mix of content. and some may prove more worthy of your time than others. Start with a radical test in which you compare a short. You can also test different topics and find out how they compare in driving business results.COm . at HubSpot we have found that ebooks perform better than webinars in converting visitors to leads.18 An introduction to A/b testing An introduction to A/b testing 19 “ What elements should I test and optimize? elements that you can optimize ” on a lanDing page: You should start your optimization process by finding what types of offers convert the most visitors into leads and which offers help you push leads down the sales funnel. CopY How should you position your offer? what messaging will entice your reader? Should you add testimonials to strengthen the visitor’s incentive? People looking for information online will pay attention to the description of your offer. will most likely be characterized by high customer close rate. but webinars do better than ebooks in converting leads to customers. you need to recognize the elements that you should spend most time testing and optimizing.COm www. at HubSpot. call-to-action and email campaigns. there are some variables you shouldn’t bother testing at all because they won’t impact clickthrough or conversion rates all that much. In fact. offers As you build landing pages. more middle-of-the-funnel offers. one-paragraph long description to longer but still valuable copy. for instance. In this section we will review both areas that are critical and some that are less critical for lead generation. bullet points and data-driven content have traditionally performed well for us. for instance. you’ve probably wondered about the elements you can test and optimize to increase conversion rate. such as consultations and demos. Consider different landing page copy that can help you drive more conversions.

HUbSPOT.20 An introduction to A/b testing An introduction to A/b testing 21 form fielDs Should your lead capture form only request an email address or should it ask for more information? offer form fielDs Inbound marketers diverge in their decision whether to place content behind a form or keep it form-free for all the world to see with no exchange of information. you have to relinquish more lead nurturing control -. Share This Ebook! www. Some argue that forms create frictions in the lead generation process.you didn’t ask for their information. Once you have a statistically significant result pointing to the variation that performed better. wHole page make iterations to the whole page that affect image placement. while others believe that forms are essential for qualifying traffic and prioritizing work for the sales organization. (With form-free content. Form fields help you qualify leads and nurture them.COm . l CopY image Share This Ebook! Landing page A/b testing enables you to evaluate how your audience reacts to different questions-what prospects are willing to answer and what information they would rather not share. you can continue optimizing through smaller tweaks. so how can you get back in touch with them unless they either bookmark your site?) As we mentioned earlier. form length and its copy. taking the entire page as the variable is the fastest way to achieve drastic results and produce a landing page that drives a lot of conversions.COm www.HUbSPOT.

website visitors read more text content than graphical content. grapHiC Share This Ebook! Share This Ebook! www. be mindful of your overall website design.22 An introduction to A/b testing An introduction to A/b testing 23 elements that you can optimize in a Call-To-aCTion: plaCemenT According to an Eyetrack III study. There has been a huge debate revolving around usage of the color red for call-to-action design. not pictures. If you need more guidance on copywriting for calls-to-action.COm . for instance. we start reading from left to right and from top to bottom.HUbSPOT. for instance. However. size while you should focus on your call-to-action text. test the ideal top and left position versus a placement in which the call-to-action is right beneath the offer description.HUbSPOT. specific. Color CopY The Eyetrack III study also showed that in ads. so make sure you do some testing. “So even the prime upper left area won’t work so well if you always put ads there. “People looking for information are looking for text.COm www. download our free ebook.” rieck writes. Create an A/b test to see if a big callto-action that adds value to the message--e. don’t forget that graphics can help convey meaning and strengthen your message. when running A/b tests to determine the highest converting color. and action-oriented. The most effective placement could also depend on the context of the rest of the page. copywriter dean rieck suggests that once your readers get used to a particular placement. Others explain that the context of the web page should determine the color. they might start ignoring the call-to-action or ad. That makes sense: in the English language. Some graphic designers argue that red can increase clickthrough rates significantly. run an A/b test to see if you get more clicks and pick a winner. Experiment with various shapes besides the standard button-like form and test how the new look affects your clickthrough rate. the CTA will naturally be large if it includes a graphic or an image that strengthens the message.g. which includes some specific tips and examples. make sure all of them fit in with your general website color scheme and avoid using patterns. a customized blog CTA--attracts more clicks than its control. Do some testing to figure out what text will make visitors more likely to click. make sure your wording is clear. Size is tightly related to the context of the page and the other characteristics of your call-to-action. the best placement of online ads is in the top and left position.” writes dean. While you want to keep the colors contrasting.

HUbSPOT. Offers. Experiment with different image and call-to-action placement and watch not only your clickthrough rates. They apply to email marketing as well. can help you drive the most conversions to one call-to-action. such as events. new offers. copy and image are certainly some of the variables you want to always keep in mind. for instance.COm . discounts and event product announcements.COm what’s the best way to lay out your email content to formaT drive the most engagement? Should your message be structured as a newsletter. newsletters can perform well in spreading the news about a few different pieces of information. so we won’t repeat them here but discuss the few new ones we haven’t mentioned so far. layout is another element that you can test and optimize for on an ongoing basis. www. dedicated sends.HUbSPOT. offer CopY Color Similar to testing formats. but also your conversions to Share This Ebook! determine which layout is most effective. on the other hand. run an A/b test to determine which format yields the results you seek to achieve. digest or a dedicated send? A/b testing can help you determine the right format of your email marketing campaigns. M laYoUT Share This Ebook! www.24 An introduction to A/b testing An introduction to A/b testing 25 grapHiC plaCemenT elements that you can optimize size in YoUr email: you must have noticed that many of the elements to test and optimize we have discussed so far overlap across different channels.

Effect of day-of-week on Clickthrough rate sUBjeCT lines Sunday monday Tuesday wednesday Thursday friday Saturday Subject lines are the part of emails that can grab your recipients’ attention immediately and convince them to open the email and read more.735 professionals 72% of respondents said that they run A/B tests to determine a winning subject line. subject lines are the part marketers seem to be testing the most about email. we have been adding the person’s signature at the end of the message. If you decide on testing email subject lines.26 An introduction to A/b testing An introduction to A/b testing 27 A/b email marketing tests that we ran at HubSpot in Timing Our Science of Timing research shows that the most clicks on email are seen on the weekend.HUbSPOT.HUbSPOT. Ensure that a change like this yields positive results for your company before you implement it. Should your prospect be sent your next email one hour later or one day later? Identify the best time to convert prospects into leads by running split tests. we started making the senders of our marketing emails be the real experts from our team.COm www. Yet only 35% find the tests effective. Share This Ebook! Share This Ebook! www. senDer So if you have been traditionally afraid of emailing on Saturdays and Sundays. try to deduct a lesson from your experiment. the optimal timing for sending marketing emails will vary by industry and even company. So you really want to get them right. 2011 showed that messages coming from a personal sender name receive higher CTr rate than messages from a generic sender name.. So instead of sending emails from Marketing Team. Interestingly enough.g. To reinforce this personalization and create consistency. Personalization is a well-known best practice in email marketing but you should still test sender names with your audience. it’s time to reconsider this practice and experiment with your timing. In marketingSherpa’s Email marketing benchmark survey of 2.COm . mention of a discount or certain formatting (e. Is it the length of the subject line. brackets or colon) that made the difference? Naturally.

target-audience@hubspot.COm .735 30% 29% 26% 22% 15% Usage Effectiveness Share This Ebook! TargeT groUp formaT laYoUT Share This Ebook! www. In fact.28 An introduction to A/b testing An introduction to A/b testing 29 who are you sending email to? Segmentation can help you get high response rates because your recipients will find the messages particularly valuable.COm www. N=2.com marketingSherpa Email marketing benchmark Survey 80% 72% 70% 60% 50% 40% 30% 20% 10% 0% 61% 50% 50% 46% 39% 35% 25% 26% 21% 22% 34% 34% 32% 41% 32% 42% Survey methodology: fielded July 2011. TargeT groUp senDer sUBjeCT Timing That will also increase the chances of them sharing your email with friends and coworkers. marketingSherpa’s survey shows that marketers find testing target audiences to be a pretty effective approach.HUbSPOT.HUbSPOT.

The positioning and the content of your copy matter. but switching “The” to “A” isn’t going to make any sort of difference.COm www. change it drastically.HUbSPOT.HUbSPOT. Some of them don’t deserve your time.COm . The switch certainly won’t hurt your marketing! a/B TesTing Case sTUDies Share This Ebook! Share This Ebook! www. mINOr TExT CHANgES d CHapTer 3 If you’re going to change your headline or subject line. like a mobile-friendly email. However.COm www. A/b testing is time-consuming and you need to be smart about the elements you decide to optimize.30 An introduction to A/b testing An introduction to A/b testing 31 elements you shouldn’t BoTHer TesTing: So far we’ve discussed some variables that you can test in order to optimize the performance of different marketing activities.HUbSPOT. gENErAL ImPrOVEmENTS If you know that something is an improvement. there is no point in testing it to see how it compares to an email that is not optimized for mobile viewing.

HubSpot was part of a live landing page optimization test in which the audience was given a control page and was asked to build a treatment page by incorporating changes that would positively affect conversion rates. More specifically. ” an example of a lanDing page TesT: This test was produced for the 2011 Landing Page Optimization Summit held by mECLAbS. we explore what a split test for a landing page. All of the examples that we discuss here have been conducted by members of the HubSpot marketing team and each test can teach you a lesson on A/B testing. what are some real-life examples of A/b testing in action? ConTrol In this section of the ebook we cover three A/b tests across different marketing channels.COm . our three case studies covered here advise you to: OPTImIzE LANdINg PAgES THAT dON’T CONVErT AT A HIgH rATE STArT yOUr OPTImIzATION PrOCESS wITH AN OffEr TEST USE THE kNOwLEdgE yOU’VE gAINEd TO ImPrOVE ExISTINg PrOCESSES Share This Ebook! Share This Ebook! www.32 An introduction to A/b testing An introduction to A/b testing 33 “ Learn from real-life A/B testing examples below. you’d probably like to see how these tests might look like. call-to-action and email looks like.HUbSPOT.COm www. Here is a screenshot of the control page: Now that we have covered some suggestions on elements that you can test and optimize.HUbSPOT. the parent company of marketingSherpa & marketingExperiments.

The conclusion we arrived at is that the audience was highly motivated by the offer itself. Share This Ebook! Share This Ebook! www. However. an example of a Call-To-aCTion TesT: The screenshot below is of a call-to-action A/b test that sought to compare two offer types. In an effort to drive a lot of traffic to this page.34 An introduction to A/b testing An introduction to A/b testing 35 The entire page was taken as a variable and was revamped. we changed the image and its placement and shortened the copy and the form. we did a heavy email marketing push. would it have a significant enough effect? In this case our control was a variation that offered the seven-day free trial and the treatment offered a 30-day free trial. The image actually illustrates what HubSpot’s homepage used to look like in 2010! Originally.HUbSPOT.3% which is a pretty negligible increase.HUbSPOT. HubSpot’s homepage offered our community a seven-day free trial.91% and the treatment converted at a rate of 48. we were curious to see if offering a longer trial period would entice more visitors to sign up. start with pages that have a low visitor-to-lead conversion rate.COm www. So when running A/b tests. It’s pretty hard to beat a page that converts 47% of the traffic. blog posts and social media promotion. ConTrol TreaTmenT Our A/b test results showed that the control converted at a rate of 47. The test was conducted using HubSpot’s landing page tools. an absolute difference of 0.COm .24%.

The 30-Day free trial won with a 99.from “maggie georgieva.326% visitor-to-free-trial conversion rate. So in 2011 we conducted a test to compare a generic “HubSpot” sender name to a personal name from someone on the marketing team. recipients also see a sender name in their inbox. who is the email coming from? This sender name can make a difference on open and clickthrough rates. 3. Such experiments will provide you with a better understanding of what prompts your visitors to convert into leads. we regularly conduct A/b tests to evaluate winning subject lines.from “HubSpot” sender name TrEATmENT .00% 1. HubSpot" www.00% 6.57% CONTrOL . an example of an email TesT: Subject lines are a critical element of email marketing. 6. comparing different offers is a great place to start.HUbSPOT.9% confidence rate and created a 110% increase in HubSpot free trials. If you want to optimize your calls-to-action (and for that matter email and landing pages).00% Type of offer can exercise a tremendous influence over lead generation efforts.00% “ 4.73% 0. HubSpot” sender name The takeaway emerging from this A/B test is that type of offer can exercise a tremendous influence over lead generation efforts.10% 7.00% Share This Ebook! www.COm ” 2. The control had a 0.COm .00% 0.00% 5. TreaTmenT but besides a subject line.00% Open rate from "HubSpot" Share This Ebook! Click Through rate from "maggie georgieva.36 An introduction to A/b testing An introduction to A/b testing 37 Results from the test showed that the 30-day free trial enticed more visitors and had a significant effect on conversion rates.96% 0.00% 7. 8. That is why we always want to make sure that we are using the right subject line when we are emailing our subscribers. They have the power to grab the attention of recipients and impact clickthrough rates to a great extent.709% visitor-to-trial conversion rate. while the treatment had a 0.HUbSPOT.

HUbSPOT.COm www. With a confidence of 99. Since our average visitor-to-lead conversion rate on landing pages is 45%.96% CTR. So how does CTr impact our number of leads generated? The treatment got 292 more clicks than the control.COm .38 An introduction to A/b testing An introduction to A/b testing 39 Our control had a 0. that means the treatment got us 131 more leads.9% we had a clear winner.HUbSPOT. CHapTer 4 rUnning & eValUaTing YoUr TesTs Share This Ebook! Share This Ebook! www.COm www. adding a personal sender name and signature to each of our messages. when your A/b tests lead to some significant results.HUbSPOT. That is great by itself but think about how valuable this lesson is when applied to our broader email marketing program.73% CTR and the treatment had a 0. Our conclusion after this A/b test was that email sent by a real person are more likely to be clicked on than email sent from the company. use the knowledge you’ve gained to improve existing processes. we took this insight and used it for the revamping of our lead nurturing campaigns.

Now that we have covered the reasons why you should run A/b tests. for statistical validity split tests need to set a cookie on each visitor to ensure the visitor sees the same variation each time they go to the tested page. we will point you to the actions you can take after running a successful test.40 An introduction to A/b testing An introduction to A/b testing 41 “ Let’s look into how you can conduct actual split tests. This is how HubSpot’s advanced landing pages and google’s website Optimizer work.HUbSPOT.HUbSPOT. let’s look into the details around their actual implementation. Share This Ebook! www. It keeps a record of the number of people who viewed each variation and number of the people who took the intended action. with the top-performing one generating 20 clicks and the lowest performing one generating five clicks.COm . it’s not enough to make a decision about whether or not your results were significant.COm In this section we will look behind the scenes of a few tools that will enable you to do A/b testing and automate the process so that you don’t have to maintain a dozen excel spreadsheets. while this information is important. what variables to optimize and how these tests might look like. ” how to do conduct a lanDing page TesT: with landing page A/b testing you have one UrL and two or more versions of the page. How do you create multiple versions of the same element you are testing? How do you split traffic randomly? How do you know if your results are statistically significant? FFF A b C HubSpot’s advanced landing pages enable you to create A/b tests and track a number of metrics to evaluate how your experiment is performing. it might inform you that each of your landing page variations was viewed 180 times. Share This Ebook! www. what’s more important. Standard landing page A/b testing tools remember which page the reader landed on and will keep showing that page to the user. for example. when you send traffic to that URL. visitors will be randomly sent to one of your variations.

USEr EffECT’S SPLIT TEST CALCULATOr ANd dECISION TOOL VISUAL wEbSITE OPTImIzEr HubSpot’s advanced landing pages will tell you when the A/B test becomes statistically significant and whether you should continue the test or stop it.42 An introduction to A/b testing An introduction to A/b testing 43 DeTermine sTaTisTiCal signifiCanCe from a high-level view. However.COm www. In this example we wanted to see if including testimonials for one of our ebooks will positively affect conversions.2% (1% better than Variation E).HUbSPOT. we are also informed that this result is not statistically significant and are encouraged to continue this test.COm . Share This Ebook! Share This Ebook! www. This led us to the conclusion that the difference between our tests is too small to affect the behavior of viewers. you can probably tell if your test results are significant or not. However. Once you’re done with the test. it’s important to test your results for statistical significance from a mathematical perspective. The tool tells us that the best choice is Variation D: it converted at 62. If the difference between the tests is very small. simply disable the variations that are losing and run with the winning page. including: SPLIT TEST CALCULATOr Y The screnshot below is an example of a landing page test that we ran at HubSpot using our advanced tool.HUbSPOT. from the data below we are able to see how many visitors each of these variations got and what the corresponding number of form submissions is. There are a number of online calculators you can use for this. it may be that the variable you tested just doesn’t influence the behavior of your viewers.

only monitoring it. If not. more radical experiment.44 An introduction to A/b testing An introduction to A/b testing 45 TesTing is aBoUT measUring oVer Time you may be tempted to check your campaign every day to try to figure out the results of your A/B test. that variable may have little bearing on your conversion rates. Effective A/b testing focuses upon continuous improvement. but there may be another element on your page that you can adjust to increase leads. ” www. It should also give you enough data to come up with an accurate conclusion. and see if that makes a difference. remember that as long as you keep working towards improvement. figure out a new iteration on your next test.HUbSPOT. TrY again You may conduct an A/B test but not find statistically significant results. you’re going in the right direction. The reason you can’t make decisions yet is that you wouldn’t be fully measuring your A/b test. it’s a good idea to wait about 15-30 days before evaluating your results.HUbSPOT. That’s like deciding you should become a pro baseball player based on a Little League game time before analyzing your results. consider testing the same variable again with different variations. t if aT firsT YoU Don’T sUCCeeD. This doesn’t mean that your A/b test has failed. then your test probably wouldn’t make a big impact on conversions. Share This Ebook! “ x Share This Ebook! Effective A/B testing focuses upon continuous improvement. make sure you have sent enough traffic to the pages.COm www. (That is often a tough task for small businesses. This will allow you to see trends as well as any issues that may have popped up. for example. This is okay. don’t be afraid to move on to another. If even after you’ve waited for 15-30 days and sent a lot of traffic to your test you haven’t seen a statistically significant result. where you hit a home run. but don’t draw any conclusions from this information just yet.) for a landing page A/b test. Measurement looks at trends over time – wait a significant amount of So how long should you wait? First.COm .

so don’t try to optimize both the call-to-action and the landing page simultaneously. you create a few variations of your CTA. ! Share This Ebook! www. make changes to one variable at a time so that you understand which element triggered the results you are seeing. but if your clickto-submission rate is surprisingly low.HUbSPOT.COm Share This Ebook! www. However. In the example below we used HubSpot’s Call-to-Action module to test the alignment between the same call-to-action and two different landing pages: remember that you should be running only one A/b test at a time. HubSpot’s call-to-action module (screenshot showed on the previous page) enables you to quickly build A/b tests and drill down to the data that matters the most to your organization. you want to be keeping an eye on your view-to-submission rate and try to optimize that.46 An introduction to A/b testing An introduction to A/b testing 47 how to do conduct a Call-To-aCTion TesT: Call-to-action split testing works pretty much the same way as landing page split testing. This result is influenced by the landing page and how well it is aligned with the call-to-action. for instance. you might look at the views-to-click rate in an effort to optimize the call-to-action. So it will be most useful to know the number of conversions each of your CTA versions drove. The goal here is to determine which call-to-action attracts the most clicks. That is why.COm . then the problem might lie with the landing page.HUbSPOT. place them on the same page and they should be displayed to visitors randomly. we mentioned earlier in this ebook that it is important to look for results further down in the sales funnel. ultimately.

The great thing about a tool like this is that it can send the winner to $ M 60% 50% 40% 30% 20% 10% 0% 30% 29% 26% 22% 15% Usage Effectiveness If we revisit the 2011 survey from marketingSherpa. In other words. significant result) and associate each email variation mailChimp.COm www. So even when it comes to improving your email marketing program.735 process and enable you to compare different elements about your email. Such tests will suggest optimization strategies that can help you achieve the most effectiveness out of your marketing emails. delivers the best results. This is a fantastic way to optimize your list and deliver the message that attracts the most attention. They randomize the list of recipients into two or more groups (you need to ensure the groups are big enough to give you a statistically to each of the groups. Share This Ebook! www.48 An introduction to A/b testing An introduction to A/b testing 49 how to do conduct an email TesT: most email providers automate the split testing marketingSherpa Email marketing benchmark Survey 80% 72% 70% 61% 50% 50% 46% 39% 35% 25% 26% 21% 22% 34% 34% 32% 41% 32% 42% Survey methodology: fielded July 2011. for this type of reporting you need to integrate your email marketing to marketing analytics. you also want to see which email is bringing in the most conversions.HUbSPOT. combined with the right landing page.HUbSPOT. identify which variation. N=2. splits your email campaign to help you find out the best subject line and the remainder of your group.COm . start by testing landing pages and put emphasis on segmentation. for instance. mailChimp and most standard email providers enable you to pick a winner based on either open rate or clickthrough rate. However. Share This Ebook! time of day to send an email. we will see that the variables which present the best results when optimized are landing page and target audience.

calls-to-action and email campaigns. walking you through processes that can improve the performance of your landing pages. you now 0 rOI Share This Ebook! www.COm ” know how to conduct split testing and what variables to isolate. and challenge your assumptions.HUbSPOT. solve problems. A/b testing will significantly increase the knowledge base of your company by allowing you to determine a set of optimized practices to boost your overall rOI. In this ebook we covered the foundations of A/b testing. Share This Ebook! www.50 An introduction to A/b testing An introduction to A/b testing 51 “ ConClUsion & aDDiTional resoUrCes These optimization efforts are directly tied to increased lead generation and customer acquisition.HUbSPOT. more importantly. A/b testing gives you the opportunity to maximize conversion rates.HUbSPOT. you also know where to start and what to spend the least time on.COm . As you begin split testing.COm www. keep in mind that all these optimization efforts are directly tied to increased lead generation and customer acquisition.

Tools HubSpot’s Advanced Landing Pages HubSpot’s Call-to-Action module Blog posts why marketers A/b Testing Shouldn’t be Limited To Small Changes 9 A/b Tests mere marketing mortals Can Use to Increase Leads 5 best Practices for website A/b Testing 10 do’s and don’ts of A/b Testing How to Determine If Your A/B Test Results Are Significant ebooks and webinars Top 7 Inspirational Email Tests what’s worth A/b Testing? Share This Ebook! www.52 An introduction to A/b testing An introduction to A/b testing 53 aDDiTional resoUrCes Check out some additional HubSpot resources on A/b testing if you are interested in learning more: rUn powerfUl a/B TesTs Sign up for HubSpot’s 30-day trial and start running easy-touse landing page and call-to-action A/b tests.com/free-Trial Share This Ebook! www.COm www.COm www.Hubspot.HUbSPOT.HUbSPOT.HUbSPOT.COm .