Designing Experiments
Section 5.2
Vocabulary of Experiments
More Vocabulary
• Placebo: a dummy treatment
• Placebo Effect: when a subject responds
favorably to a placebo
• Control Group: a group that provides a
standard for comparison to evaluate the
effectiveness of a treatment; often given the
placebo.
Blinding
• Single-blind experiments: the subjects do not
know which treatment (or placebo) the are
receiving.
• Double-blind: if both the subjects and the
researchers don’t know which treatment each
subject receives.
The Three Key Principles:
• The three key principles of experimental
design are:
– Control
– Replication
– Randomization
CRR
Who’s Really in Control?
• Don’t confuse Control with Control Group.
• Control refers to the overall effort to
minimize variability in the way the
experimental units are obtained and treated.
The Heart of the Matter
• We carry out experiments to see if there is a difference
in the responses of those who received the treatment and
those who didn’t.
• We hope to see a difference in the responses so large
that it is unlikely just to happen by chance variation.
• We can use the laws of probability to learn if the
differences in treatment effects are larger than we would
expect to see if only chance were operating.
• If they are, we call them statistically significant.
Statistical Significance
An Aspirin a Day…
• You often see the phrase “statistically significant” in
reports of investigations in many fields of study. That
means that investigators found good evidence for the
effect they were seeing.
• For example, the Physicians’ Health Study reported
statistically significant evidence that aspirin reduces
the number of heart attacks compared to a placebo.
Completely Randomized Design
(CRD)
• When all experimental units are randomly allocated
among all the treatments (or treatments to the units),
it is said to be a completely randomized design.
• CRDs can compare any number of treatments. The
treatments can be formed by levels of a single factor
or by more than one factor.
You Blockhead!
• Suppose a fitness instructor believes that a certain exercise
regimen will increase upper-body strength. He recruits
students to test his theory by having them do as many push-
ups as they can after they complete the training.
• We would expect some variability due to natural differences in
strength. We try to control for these inherent differences by
placing subjects into groups of similar individuals. Because
we know that women have less upper-body strength than men,
we would put them into different groups, or blocks.
• By separating subjects by gender, we can reduce the effect of
variation in strength on the number of push-ups.
Randomized Block Design (RBD)
• A block is a group of experimental units that are
known to be similar in some way that is expected to
systematically affect the response to the treatment.
• In a randomized block design, the random
assignment of units to treatments (or treatments to
units) is carried out separately within each block.
The Mantra
• Blocking allows us to draw separate conclusions about each block.
Blocking also allows more precise overall conclusions because the
systematic differences between the blocks can be removed when
we study the overall effects of the treatments.
• A good experimenter will form blocks based on the most
important, but unavoidable, sources of variability among the
experimental units. Randomization will then average out the
effects of the remaining variation and allow an unbiased
comparison of the treatments.
• The mantra: Control what you can, block on what you can’t
control, and randomize the rest.
Can You Elaborate?
• CRDs are the simplest experimental designs,
but they are often inferior to more elaborate
designs. In particular, matching subjects in
various ways can produce more precise results
than simple randomization.
• The simplest form of matching is called a
matched-pairs design.
A Better Comparison?
• To compare two treatments (or a treatment and a
placebo), you can use experimental units that are as
alike as possible. The idea is that matched units are
more similar than unmatched units, so comparing
responses in a number of pairs is more efficient that
comparing the responses of a group of randomly
assigned subjects.
• Randomization is still important though! The order in
which the treatment is imposed (or the assigning of a
treatment and a placebo to each unit in a pair) should
still be randomly generated.
Caution!
• Remember not to generalize!
• Statistical analysis of an experiment cannot tell
us how far the results will generalize to other
setting or individuals.
Outlines of Experiments
Jasper Leigh
