Module #5: Experimental Design

(Reading: Whitlock and Schluter Ch.14 & Interleafs 5, 6)

Crafting well designed experiments to test well thought out hypotheses is a

really important skill as a scientist (and one of the most fun aspects of
science!). Sadly, it is not always as easy as you might think.
The first major distinction when we think about designing a study is the
difference between:
1. Observational studies -- “where nature assigns individuals to
treatments” p. 423
2. Experimental studies – “where the researcher assigns individuals to
treatments” p. 423

When examining the results of a study, why is it important to consider if it

was an experimental or observational study? How does the type of study
impact what you can infer from the study (for both experimental and
observational studies)?

Is one type of study better than the other? No

Experimental Studies
The major advantage of experimental studies is that we are able to minimize
the impact of confounding variables on the outcome of the experiment.
Example of a confounding principle
Confounding variable = heat
> ppl heat more ice cream when its hot
violent > there is no association just a correlation
pattern
crimes
> it masks the cause and realtionship btw
two variabkles
ice cream consumption
Confounding variables -- are variables that mask or distort the causal
relationship between measured variables in the study. p. 424
What aspect of an experimental study minimizes the impact of confounding
variables? Explain.
Randomization = by randomly assigning to treatments the association btw the confounding variables is broken, the
effect of the treatment can be broken

A disadvantage of experiments are experimental artifacts.

Experimental artifact – is a bias in a measurement produced by unintended
eg. sleep studies = you go to labs and sleep where the scientists is
consequences of experimental procedures. watching you and that might nit be the natural sleeping
environment as required by the study
What can be done to minimize experimental artifacts?
>> a control
>> try to replicate the natural system as much as possible
Two Goals of Experiments (Reducing Bias and decreasing sampling error)

I. Reducing Bias
1. Simultaneous control group – subjects who are
treated like all of the experimental subjects, except that the
control group does not receive the treatment p. 428

Why do we need controls?

>> several diff reasons
1. for the effect of time
2. measure the experimental artifacts
3. determine any placebo affects (must be above and beyond the placebo effect)

Discuss the following scenarios:

i. A student starts to get a cold and so begins taking Echinacea. After 2
days, the student’s cold is gone. The Echinacea worked!
 What might be wrong with the conclusion that the Echinacea
worked?
>> Its only one person
>> cold might just go away naturally after one day

What would be a proper control for this ‘experiment’?

> have a group of people and randomly assign them to control and treatment groups

ii. Enclosures are set up in lake to examine the behaviour of fish

exposed to antidepressants (often found in the output from
wastewater treatment plants). Half the enclosures are randomly
selected to receive the antidepressant treatment. Rainbow trout in the
treatment group are caught, injected with the dosage of antidepressant
typically found in wastewater. The control groups are left untreated.
Behavioural observations on both groups are performed. The behaviour
of the treatment fish differs significantly from the control fish due to
the effect of antidepressants.
What might be wrong with the conclusion?
> leaving control group left on their own with no intervention

Design a proper control for this experiment.

2. Randomization – is the random assignment of

treatments to units in an experimental study p. 429.

If treatments are truly randomly assigned to experimental units

(individuals), then any patterns that may have existed in the sample of
individuals used for the experiment are randomly distributed between
control and treatment groups and any impacts of this confounding variable
are removed from the effect of the treatment.

*randomization breaks the association between possible confounding

variables and the explanatory variable*
Best efforts should be made to randomly assign treatments.
What are some possible methods you can use to randomly assign
treatments to individuals?
.> using some kind of random number generator
> flipping a coin is perfectly okay too

3. Blinding – is the process of concealing information

from participants (sometimes including researchers) about
which individuals receive which treatment.

Single blind: participants are unaware of the treatment they have

been assigned
Double blind: researcher and participants are unaware of which
subjects are receiving which treatment
Why is blinding so important?
> placebo effect
> also researchers can subconsciously affect the results of the study

What might be some risks with only having a single blind?

 Is it easier to think about blinding in a clinical trial, but how might blinding
be applied when you are conducting other types of research?

II. Reducing Sampling Error

Not only do we want to eliminate bias, but we also want as precise of

estimates as possible and to reduce background noise so that we can see
the effect of our treatment.
1. Replication – this is essential to every experiment. You
must have application of treatment to multiple,
independent experimental units.

Replication seems like an obvious aspect of experimental design, however,

many experiments are plagued by pseudo-replication. Correctly identifying
independent replicates can be challenging.
Consider the following scenarios. Are these independent replicates?
> zooplankton in a lake and over the summer 20 samples of the zooplankton

N=26
zoo
plankton

> another way is you go to forest and transect and measure every tree along the transect
n=10
- whats wrong, they are spatially close together so they are not independent of each other so thats not a random sample
(pseudoreplication)
 2. Blocking -- helps to account for extraneous variation
by dividing experimental units into groups. A paired t-test is
the simplest form of blocking.

Let’s say you are interested in examining the effect of CO2 fertilization on a
prairie grassland. You have three treatments (control (ambient CO2 level),
low CO2 and high CO2). Sketch two different experimental designs: 1)
Random and Independent experimental groups 2) Randomized block
design. Be sure to describe how you would determine which individual unit
gets assigned to which treatment.
treatments Control, Low, High 5 replicates

Random Blocked

> grid over the study area

in a random order
C L H
L C H

3. Balanced – a balanced design is one that has equal

number of observations in each treatment.

Balance is important because it minimizes the SE associated with the

treatments.
Allows the equally precise measures of the means associated with the
different treatments. Larger samples are always better, but if the allocation
to sampling is fixed, it is best to distribute the number of observations you
will be measuring equally between the different treatment groups.

Designing Powerful Experiments: Some tips

1. Decrease the number of treatment levels in favour of increasing the
sample size within each treatment group.
5 level = 4 reps

4 level = 5 reps (more power)

2. Use extreme treatments

> better to test at a big amount rather than smaller amount
> would help to know if the treatment is working
In a clinical trial for a new treatment for Multiple Sclerosis, patients
undergo surgery designed to widen the veins in the neck. Would it be a
mistake to have a control group that is left with no intervention at all
for the duration of the clinical trial?
a. Yes
b. No

ARTICLE

Control Group –

Randomization –

Blinding --