5.
2 Designing
Experiments
November 8, 2011
Experimental Units, Subjects,
Treatment
Experimental Units: the individuals on
which the experiment is done.
Subjects: if the experimental units are
people, they are subjects.
Treatment: the specific experimental
condition applied to the units.
Factor: explanatory variables in an
experiment
Effects of Class Size
Example 5.13
Do smaller class sizes in elementary school really
benefit students in areas such as scores on standardized
tests, staying in school, and going to college? We might
do an observational study that compares students who
happened to be in smaller and larger classes in their
early school years. Small classes are expensive, so
they are more common in schools that serve richer
communities. Students in small classes tend to also
have other advantages; their schools have more
resources, their parents are better educated, and so on.
Confounding makes it impossible to isolate the effects of
small classes.
Effects of Class Size
The Tennessee STAR program was an experiment on the
effects of class size. It has been called “one of the most
important educational investigations ever carried out.” The
subjects were 6385 students who were beginning kindergarten.
Each student was assigned to one of three treatments: regular
class (22 to 25 students) with one teacher, regular class with
one teacher and a teacher’s aid, and small class (13 to 17
students). These treatments are levels of a single factor, type
of class. The students stayed in the same type of class for four
years, then all returned to regular classes. In later years,
students from the small classes had higher scores on
standardized tests, were less likely to fail a grade, had better
high school grades, and so on. The benefits of small classes
were greatest for minority students.
Control
Treatment Observe Response
Laboratory experiments in science and
engineering
Simple design with only a single treatment
Control group: a group of patients who
receive a fake treatment, i.e. sugar pill.
Placebo effect: when patients respond to a
“dummy treatment”
Replication
Use enough subjects to reduce chance
variation
Would the class size study have worked if there
were only 50 subjects?
Randomization
The rule used to assign the experimental
units to the treatments.
Comparison of the effects of several
treatments is valid only when all treatments
are applied to similar groups of experimental
units.
Cell Phones and Driving
Example 5.17
Does talking on a hands-free cell phone
distract drivers? Undergraduate students
“drove” in a high –fidelity driving simulator
equipped with a hands-free cell phone. The
car ahead breaks: how quickly does the
subject respond? Twenty students (the
control group) simply drove. Another 20
students (the experimental group) talked on
the phone while driving.
Cell Phones and Driving
This experiment has a single factor (cell
phone use) with two levels. The researchers
must divide the 40 student subjects into two
groups of 20. To do this in a completely
unbiased fashion, put the names in a hat, mix
them up, and draw 20. These students form
the experimental group, and the 20 remaining
make up the control group.
Randomized Comparative
Experiments
Randomization produces two groups of subjects
that we expect to be similar in all respects
before the treatments are applied.
Comparative design helps ensure that
influences other than the cell phone operate
equally on both groups.
Therefore, differences in average brake reaction
time must be due either to talking on the cell
phone or to the play of chance in the random
assignment of subjects of the two groups.
Principles of Experimental
Design
1. Control the effects of lurking variables on
the response, most simply by comparing two
or more treatments.
2. Replicate each treatment on many units to
reduce chance variation in the results.
3. Randomize – use impersonal chance to
assign experimental units to treatments.
Statistical Significance
An observed effect so large that it would
rarely occur by chance
Completely Randomized
Design
When all experimental units are allocated at
random among all treatments
Homework
Pg. 357 #5.33-35
Pg. 364 #5.39-40
Block Design
The random assignment of units to
treatments is carried out separately within
each block.
A block is a group of experimental units or
subjects that are known before the experiment to
be similar in some way that is expected to
systematically affect the response to the
treatments.
Another form of control
Example 5.20 Comparing
Cancer Therapies
The progress of a type of cancer differs in women
and men. A clinical experiment to compare three
therapies for this cancer therefore treats gender as
a blocking variable. Two separate randomizations
are done, one assigning the female subjects to the
treatments and the other assigning the male
subjects. The figure on the next slide outlines the
design for the experiment. Note that there is no
randomization involved in making up the blocks.
They are groups of subjects that differ in some way
(gender in this case) that is apparent before the
experiment begins.b
Example 5.20
Figure 5.7
Blocks Important Info:
Blocks allow us to draw separate conclusions
about each block.
Allows more precise conclusions because
differences (i.e. between men and women)
are removed.
A wise experimenter will form blocks
based on the most important unavoidable
sources of variability among experimental
units.
Control what you can, block what you can’t,
and randomize the rest.
Matched Pairs Design
Matching subjects in various ways can
produce more precise results than simple
randomization.
Matched pairs design: the simplest use of
matching which compares just two
treatments.
Randomization is still important!
Example 5.23 Cell Phones and
Driving, II
Example 5.17 describes an experiment on
the effects of talking on a cell phone while
driving. The experiment compared two
treatments: driving in a simulator and driving
while talking on a hands-free phone. The
response variable is time the driver takes to
apply the brake when the car in front brakes
suddenly. This example is a completely
randomized design. Subjects differ in driving
skill and reaction times. The completely
randomized design relies on chance to create
two similar groups of subjects.
Example 5.23
In fact, the experimenters used a matched pairs
design in which all subjects drove both with and
without using the cell phone. They compared
each individual’s reaction times with and without
the phone. If all subjects drove first with the
phone and then without it, the effect of talking on
the cell phone would be confounded with the
fact that this is the first run in the simulator. The
proper procedure requires that all subjects first
be trained in using the simulator, that the order
in which a subject drives with and without the
phone be random, and that the two drives be on
separate days to reduce carryover effects.
Please note:
Matched pairs design is a form of block
design.
It simply compares two treatments.
Cautions about
Experimentation
Good experiments require careful attention to
detail.
Double-Blind Experiment: Neither the
subjects nor those who measure the
response variable know which treatment a
subject received.
helps eliminate placebo effect
helps eliminate doctor bias, i.e. a doctor doesn’t
think “just a placebo” can effect a patient.
Cautions about
Experimentation
Many (maybe most??) experiments have
some weaknesses in detail.
The environment of an experiment can
influence the outcomes in unexpected ways.
Lack of Realism: the subjects or treatments
or setting of an experiment may not
realistically duplicate the conditions we really
want to study.
Example 5.25 Placebo
Cigarettes?
A study of the effects of marijuana recruited
young men who used marijuana. Some were
randomly assigned to smoke marijuana
cigarettes, while others given placebo
cigarettes. This failed: the control group
recognized that their cigarettes were phony
and complained loudly. It may be quite
common for blindness to fail because the
subjects can tell which treatment they are
receiving.
Homework
Pg. 372 #5.47
5.2 Quiz Due Thursday (Quiz Grade!)
