• Example question 2: Temperature and soil respiration

• You want to know how temperature affects soil respiration. Specifically, you ask how
increased air temperature near the soil surface affects the amount of carbon dioxide
(CO2) respired from the soil.

• To translate your research question into an experimental hypothesis, you need to

define the main variables and make predictions about how they are related.

Step 2: Write your hypothesis

• Now that you have a strong conceptual understanding of the system you are studying,
you should be able to write a specific, testable hypothesis that addresses your research
question.

• How you manipulate the independent variable can affect the experiment’s external
validity – that is, the extent to which the results can be generalized and applied to the
broader world.

Step 3: Design your experimental

treatments
• First, you may need to decide how widely to vary your independent variable.

• Soil-warming experiment

• You can choose to increase air temperature:

• just slightly above the natural range for your study region.

• over a wider range of temperatures to mimic future warming.

• over an extreme range that is beyond any possible natural variation.

• Second, you may need to choose how finely to vary your independent variable.
Sometimes this choice is made for you by your experimental system, but often you
will need to decide, and this will affect how much you can infer from your results.

• Phone-use experiment

• You can choose to treat phone use as:

• a categorical variable: either as binary (yes/no) or as levels of a factor (no phone use,
low phone use, high phone use).

• a continuous variable (minutes of phone use measured every night).

 Step 4: Assign your subjects to
treatment groups
• How you apply your experimental treatments to your test subjects is crucial for
obtaining valid and reliable results.

• First, you need to consider the study size: how many individuals will be included in
the experiment? In general, the more subjects you include, the greater your
experiment’s statistical power, which determines how much confidence you can have
in your results.

• Then you need to randomly assign your subjects to treatment groups. Each group
receives a different level of the treatment (e.g. no phone use, low phone use, high
phone use).

• You should also include a control group, which receives no treatment. The control
group tells us what would have happened to your test subjects without any
experimental intervention.

• When assigning your subjects to groups, there are two main choices you need to
make:

• completely randomized design vs a randomized block design.

• An independent measures design vs a repeated measures design.

• Randomization

• An experiment can be completely randomized or randomized within blocks (aka

strata):

• In a completely randomized design, every subject is assigned to a treatment group at

random.

• In a randomized block design (aka stratified random design), subjects are first
grouped according to a characteristic they share, and then randomly assigned to
treatments within those groups.
 The 7 steps of experimental
design
• Terms in this set (6)

• Identify the problem or question.

• Form a hypothesis or solution to the problem.

• Design the experiment to be used to test your hypothesis.

• Carry out experiment.

• Analyze data and observations.

• State conclusion.

What's a Marketing Experiment?

A marketing experiment is a form of market research. It's a test
organizations run to discover possible marketing avenues that will improve
a campaign.

For instance, a marketing team might create and send emails to a

small segment of their overall readership to gauge engagement rates,
before adding them into a campaign. Additionally, they might A/B test the
design of these emails. In this example, the team is creating a hypothesis
(that a certain email design will help promote their campaign) and testing
the hypothesis in a marketing experiment.
It's important to note that a marketing experiment isn't synonymous with a
marketing test. Marketing experiments are done for the purpose of
discovery, while a test confirms theories.

Ultimately, a marketing experiment can help you ensure your campaign or

strategy will be effective. Next, let's dive into how to conduct a marketing
experiment.

How to Conduct a Marketing Experiment

Performing a marketing experiment lets you try out different methods of
running a campaign to see which one will perform the best. It involves
doing background research, structuring the experiment, and analyzing the
results.

Now, let's go through the five steps necessary to conduct a marketing

experiment.

1. Make a hypothesis.

Hypotheses aren't just related to science projects. When conducting a

marketing experiment, the first step is to make a hypothesis you're curious
to test.

Let's say you want to make a marketing email that will improve
engagement rates. A good hypothesis for this might be, "Making an email
with emojis in both the subject line and copy will increase our engagement
rates by at least 25%." This is a good hypothesis because you can prove or
disprove it, it isn't subjective, and it has a clear measurement of
achievement.
2. Collect research.

After creating your hypothesis, begin to gather research. Doing this will give
you background knowledge about experiments that have already been
conducted and get an idea of possible outcomes.

Researching your experiment can help you modify your hypothesis if

needed. If your hypothesis is, "Making an email with an emojis in the
subject line and copy will increase our engagement rates by at least 25%,"
and research on trends in your audience on email subject lines show that to
be true, you know you have a solid hypothesis. However, if other
companies in your industry haven't seen success from emojis in emails,
you might want to reconsider.

3. Choose measurement metrics.

Once you've collected the research, you can choose which avenue you will
take and what metrics to measure.
For instance, maybe you will run an A/B test. This method will allow you to
measure the results of two different emails, and figure out which email
performs better with your target audience..

For a marketing email test, consider measuring impressions, reach,

conversion rate, or clickthrough rate (CTR). These email metrics can let
you know how many people are receiving, opening, and reading your
emails, and will help you analyze the results of your hypothesis.

4. Create and execute the experiment.

Now it's time to create and perform the experiment. If you're creating
an A/B test to prove your hypothesis about emojis in emails, then you'll
want to create two emails -- one with a plain text subject line, and an
identical email with 1-2 emojis added to the subject line. Try to only
make slight variations between emails A and B to ensure accuracy.

When you're finished designing the experiment, come up with a timeline,

and decide how you'll monitor the results. That way, when conducting the
A/B test, you'll be prepared to swiftly figure out which email performed
better.

Finally, choose your recipients and conduct the experiment. Next, you'll
analyze your results.

5. Analyze the results.

Once you've run the experiment, collect and analyze the results. Use the
metrics you've decided upon in the second step and conclude if your
hypothesis was correct or not.

The prime indicators for success will be the metrics you chose to focus on.

For instance, for the marketing email example, did engagement numbers
appear higher? If the CTR, impressions, and click-to-open rates are at or
higher than the 25% goal, the experiment would be considered one where
the hypothesis was accepted.

Now that you know how to conduct a marketing experiment, let's go over a
few different ways to run them.
Types of Marketing Experiments
There are many types of marketing experiments you can conduct with your
team. These tests will help you determine how aspects of your campaign
will perform before you roll out the campaign as a whole.

1. A/B Testing

A/B testing is a popular marketing experiment in which two versions of a

webpage, email, or social post are presented to an audience (randomly
divided in half). This test determines which version performs better with
your audience.

HubSpot's email tool offers an A/B test feature for Professional and
Enterprise users. Alternatively, check out 8 of the Best A/B Testing Tools
for 2019 for options of other tools to help you perform A/B tests.

This method is useful because you can better understand the preferences
of users who will be using your product.

2. Different CTAs

Experimenting with different CTAs can improve the number of people who
engage with your content. For instance, instead of using "Buy now!" to pull
customers in, why not try, "Learn more?"

You can also test different colors of CTAs as opposed to copy. Another
CTA factor that I've been seeing around are ones that are animated.

To learn more about different types of CTAs, check out 8 Types of CTAs
You Should Absolutely Try on Your Blog.

For a CTA-related marketing experiment, you'll want to either use PPC ads
or landing pages to insert your CTA . From there, measure relevant metrics
based on your hypothesis and design of the button.

3. Animated Ads

As a big purveyor of GIFs in the workplace, animating ads are a great way
to catch the attention of potential customers. Animating ads don't
necessarily mean using GIFs -- you might also try small videos or ads with
multiple cards, which can catch the attention of web browsers.

This Instagram ad from Buffer, above, uses multimedia to make their post
pop. If you're testing out PPC advertising, try diversifying those ads to
capture the interest of more audiences. Additionally, you might run different
types of copy with your ads to see which language compels your audience
to click.
LINK VIDEO:
https://www.instagram.com/p/B2hLPCtnKP1/?utm_source=ig_web_button_share_sheet

4. Social Media Platforms

Is there a social media site you're not using? For instance, lifestyle brands
might prioritize Twitter and Instagram, but implementing Pinterest opens
the door for an untapped audience.

You might consider testing which hashtags or visuals you use on certain
social media sites to see how well they perform. The more you use certain
social platforms, the more you can iterate based on what your audience is
engaging with the most.

5. Experiment Globally

If you post on an Eastern Time Zone (ET) schedule, run an experiment that
involves Pacific or Central Time Zones.

You might even use your social media analytics to determine which
countries or regions you should focus on -- for instance, my Twitter
Analytics, below, demonstrates where most of my audience resides. If,
alternatively, I saw most of my audience came from India, I might need to
alter my social strategy to ensure I catered to India's Time Zone, as well.
 When experimenting with different time zones, consider making content
specific to the audience you're trying to reach. If you're trying to reach global
audiences, why not post something in a few different languages? Alternatively,
if you have international offices, you might spotlight different employees from
your offices all over the globe.
Ultimately, marketing experiments are a cost-effective way to get a picture
of how new content ideas will work in your next campaign, which is critical
for ensuring you continue to delight your audience. For more new content
ideas, check out our ultimate round-up here.

Key Takeaways

• Experiments can be conducted using either between-subjects or within-subjects designs. Deciding

which to use in a particular situation requires careful consideration of the pros and cons of each
approach.
• Random assignment to conditions in between-subjects experiments or to orders of conditions in
within-subjects experiments is a fundamental element of experimental research. Its purpose is to
control extraneous variables so that they do not become confounding variables.
• Experimental research on the effectiveness of a treatment requires both a treatment condition and a
control condition, which can be a no-treatment control condition, a placebo control condition, or a
waitlist control condition. Experimental treatments can also be compared with the best available
alternative.
 Exercises

1. Discussion: For each of the following topics, list the pros and cons of a between-subjects and
within-subjects design and decide which would be better.
1. You want to test the relative effectiveness of two training programs for running a marathon.
2. Using photographs of people as stimuli, you want to see if smiling people are perceived as more
intelligent than people who are not smiling.
3. In a field experiment, you want to see if the way a panhandler is dressed (neatly vs. sloppily)
affects whether or not passersby give him any money.
4. You want to see if concrete nouns (e.g., dog) are recalled better than abstract nouns (e.g., truth).
2. Discussion: Imagine that an experiment shows that participants who receive psychodynamic therapy
for a dog phobia improve more than participants in a no-treatment control group. Explain a
fundamental problem with this research design and at least two ways that it might be corrected.

1. Price, D. D., Finniss, D. G., & Benedetti, F. (2008). A comprehensive review of

the placebo effect: Recent advances and current thought. Annual Review of
Psychology, 59, 565–590. ↵
2. Shapiro, A. K., & Shapiro, E. (1999). The powerful placebo: From ancient
priest to modern physician. Baltimore, MD: Johns Hopkins University Press. ↵
3. Moseley, J. B., O’Malley, K., Petersen, N. J., Menke, T. J., Brody, B. A.,
Kuykendall, D. H., … Wray, N. P. (2002). A controlled trial of arthroscopic
surgery for osteoarthritis of the knee. The New England Journal of Medicine,
347, 81–88. ↵
4. Birnbaum, M.H. (1999). How to show that 9>221: Collect judgments in a
between-subjects design. Psychological Methods, 4(3), 243-249. ↵