Chapter 4

Section 4.2
Experiments
Warm up
Identify the sources of bias:
1. Before the presidential election of 1936, FDR
against Republican ALF Landon, the magazine
Literary Digest predicting Landon winning the
election in a 3-to-2 victory. A survey of 10 million
people. George Gallup surveyed only 50,000
people and predicted that Roosevelt would win.
The Digest’s survey came from magazine
subscribers, car owners, telephone directories, etc.
Undercoverage – since the Digest’s survey
comes from car owners, etc., the people
selected were mostly from high-income
families and thus mostly Republican!
(other answers are possible)
2. Suppose that you want to estimate the total
amount of money spent by students on textbooks
each semester at SMU. You collect register
receipts for students as they leave the bookstore
during lunch one day. Conveniencecollect
sampling – easy way to
data
or
Undercoverage – students who buy
books from on-line bookstores are
excluded.
3. To find the average value of a home in Memorial,
one averages the price of homes that are listed for
sale with a realtor. Undercoverage – leaves out homes
that are not for sale or homes
that are listed with different
realtors.
(other answers are possible)
 Observational Studies vs
Experiments
Observational Study Experiment
• Observes individual and • Deliberately imposes some
measures variables of treatment on individual in
interest but does not order to observe their
attempt to influence the responses
responses • Only source of full convincing
• Sample survey data
• Poor way to show the • When our goal is to
effect of an intervention understand cause and effect,
experiments are the only
source of fully convincing
data

4
1. In 2001, a report in the journal of the American Cancer
Institute indicated that women who work nights have a 60%
greater risk of developing breast cancer. Researchers based
these findings on the work histories of 763 women with breast
cancer and 741 women without breast cancer.
What’s the design? (observational study or experiment)
Observational Study

2. Scientists want to investigate the effectiveness of an herbal

compound to treat common cold. They exposed each subject to
a cold virus, then gave him/her either the herbal compound or a
sugar solution. Several days later, they assessed the patient’s
condition, using a scale of 0 to 5, a cold severity scale, they
found no evidence of benefits associated with the compound.
What’s the design? Experiment
Example (p. 235):
Should women take hormones such as estrogen after menopause,
when natural production of these hormones ends? In 1992, several
major medical organizations said “Yes.” Women who took hormones
seemed to reduce their risk of a heart attack by 35% to 50%. The
risks of taking hormones appeared small compared with the benefits.

To get convincing data on the link between hormone replacement and

heart attacks, we should do an experiment. Randomly assign women
to either hormone replacement pills or to placebo pills. By 2002,
several experiments with women of different ages agreed that
hormone replacement does not reduce the risk of heart attacks.

Response Variable? Explanatory Variable?

Whether the Whether or not
woman had a the woman took
heart attack hormones
 Observational Study vs. Experiment
Observational studies of the effect of an explanatory variable on a
response variable often fail because of confounding between the
explanatory variable and one or more other variables.

Well-designed experiments take steps to prevent confounding.

Confounding occurs when two variables are associated in

such a way that their effects on a response variable cannot
be distinguished from each other.

When identifying a possible confounding variable you MUST explain how the
variable is associated with the explanatory AND affects the response variable.

Name a confounding variable from the previous example:

number of doctor visits per year- women who chose to take hormones
visited their doctors more than women who didn’t take hormones and
women who had heart attacks probably didn’t go to the doctors as often as
women who didn’t have heart attacks.
 The Language of Experiments
An experiment is a statistical study in which we actually do something
(a treatment) to people, animals, or objects (the experimental units) to
observe the response. Here is the basic vocabulary of experiments.

A specific condition applied to the individuals in an experiment is

called a treatment. If an experiment has several explanatory
variables, a treatment is a combination of specific values of these
variables.

The experimental units are the smallest collection of individuals

to which treatments are applied. When the units are human
beings, they often are called subjects.
Example (p. 237)
Researchers at UNC were concerned about the increasing dropout rate in the
state’s high schools, especially for low-income students. Surveys of recent dropouts
revealed that many of these students had started to lose interest during middle
school. They said they saw little connection between what they were studying in
school and their future plans. To change this perception, researchers developed a
program called CareerStart. The big idea of the program is that teachers show
students how the topics they learn get used in specific careers.

To test the effectiveness of CareerStart , the researchers recruited 14 middle

schools in Forsyth County to participate in an experiment. Seven of the schools,
determined at random, used CareerStart along with the distric’s standard
curriculum. The other seven schools just followed the standard curriculum.
Researchers followed both groups of students for several years, collecting data on
students’ attendance, behavior, standardized test scores, level of engagement in
school, and whether or not the students graduated from high school.

Results: Students at schools that used CareerStart generally had better attendance
and fewer discipline problems, earned higher test scores, reported greater
engagement in their classes, and were more likely to graduate.

Identify the experimental units, explanatory and response variables, and the
treatments in the CareerStart experiment.
Experimental Units- 14 middle schools in Forsyth County,
NC

Explanatory variable- whether the school used the

CareerStart program with its students

Several response variables were measured- test scores,

attendance, behavior, student engagement, and graduation
rates.

This experiment compares two treatments: (1) the

standard middle school curriculum and (2) the standard
curriculum plus CareerStart
A study published in the New England Journal of Medicine
(March 11, 2010) compared two medicines to treat head lice:
an oral medication called ivermectin and a topical lotion
containing malathion. Researchers studied 812 people in 376
households in seven areas around the world. Of the 185
households randomly assigned to ivermectin, 171 were free
from head lice after 2 weeks compared with only 151 of the
191 households randomly assigned to malathion.
Problem: Identify the experimental units, explanatory and
response variables, and the treatments in this experiment.

The experimental units are the 376 households, not the 812 people,
because the treatments were assigned to entire households, not separately
to individuals within the household. The explanatory variable is type of
medication, and the response variable is whether the household was lice-
free. The treatments were ivermectin and malathion.
 Experiments
Many students regularly consume caffeine to help them
stay alert. So, it seems plausible that taking caffeine might
increase an individual’s pulse rate. Is this true? One way to
investigate this claim is to ask volunteers to measure their
pulse rates, drink some cola with caffeine, measure their
pulse rates again after 10 minutes, and calculate the
increase in pulse rate.
Unfortunately, even if the pulse rate of every student went up, we
couldn’t attribute the increase to caffeine. Perhaps the excitement of
being in an experiment made their pulse rates increase. Perhaps it
was the sugar in the cola and not the caffeine. Perhaps their teacher
told them a funny joke during the 10-minute waiting period and
made everyone laugh! In other words, there are many other variables
that are potentially confounded with taking caffeine.
 How to Experiment Badly
Many laboratory experiments use a design like the one in the
caffeine example:

Measure
Experimental Units Treatment
Response

In the lab environment, simple designs often work well E.g: Finding
the strength of a beam by subjecting a load to the beam and observing
its deflection

Field experiments and experiments with animals or people deal with

more variable conditions. E.g. Gastric Freezing for treating ulcers

Outside the lab, badly designed experiments often yield worthless

results because of confounding.
 How to Experiment Well
The remedy for confounding is to perform a comparative
experiment in which some units receive one treatment and
similar units receive another. Most well designed
experiments compare two or more treatments.

Comparison alone isn’t enough, if the treatments are given

to groups that differ greatly, bias will result. The solution to
the problem of bias is random assignment.

In an experiment, random assignment means that

experimental units are assigned to treatments using a
chance process.
 Suppose you have a class of 30 students who volunteer to be
subjects in the caffeine experiment described earlier.
Problem: Explain how you would randomly assign 15 students to
each of the two treatments:
(a) Using 30 identical slips of paper
(b) Using technology
(c) Using Table D
Solution:
(a) Using 30 identical slips of paper, write A on 15 pieces of paper and B on the other 15. Mix
them thoroughly in a hat and have each student select one slip of paper. Then ask each
student who received an A to drink the cola with caffeine and each student who received a B
to drink the cola without caffeine.
(b) Number the students from 1 to 30. Use randInt(1,30) to select 15 different numbers from
1 to 30. These students will drink the cola with caffeine and the remaining 15 will drink the
cola without caffeine.
(c) Number the students from 01 to 30. Use a line from Table D and read two-digit numbers
moving from left to right. The first 15 different numbers from 01 to 30 will identify the
students who will drink the cola with caffeine. The remaining 15 students will drink the cola
without caffeine.
 Warm up
Identify the experimental units, the explanatory and response
variables and the treatments.
1. You can use Voice over Internet Protocol (VoIP) to make
long-distance calls over the Internet. One of the most
popular VoIP services is Skype. How will the appearance
of ads during calls affect the use of this service?
Researchers design an experiment to find out. They
recruit 300 people who have not used Skype before to
participate. Some people get the current version of Skype
with no ads. Others see ads whenever they make calls.
The researchers are interested in frequency and length of
phone calls.
Experimental Units: 300 people who haven’t used Skype
before
Explanatory Variable: whether ads are present
Response Variables: Frequency and length of phone calls
Treatments: no ads shown during phone calls and ads shown
during phone calls
 Warm up
2. Most American adolescents don’t eat well and don’t exercise
enough. Can middle schools increase physical activity among
their students? Can they persuade students to eat better?
Investigators designed a “physical activity intervention” to
increase activity in physical education classes and during leisure
periods throughout the school day. They also designed a
“nutrition intervention” that improved school lunches and
offered ideas for healthy home-packed lunches. Each
participating school was randomly assigned to one of the
interventions, both interventions, or no intervention. The
investigators observed physical activity and lunchtime
consumption of fat.
Experimental Units: middle schools
Explanatory Variable: whether a physical activity program was offered
and whether a nutrition program was offered
Response Variables: physical activity and lunchtime consumption of fat
Treatments: (1) activity intervention only; (2) nutrition intervention
only; (3) both interventions; (4) neither intervention
Principles of Experimental Design
Principles of Experimental Design
The basic principles for designing experiments are as follows:
1. Comparison. Use a design that compares two or more treatments.
2. Random assignment. Use chance to assign experimental units to
treatments. Doing so helps create roughly equivalent groups of
experimental units by balancing the effects of other variables among
the treatment groups.
3. Control. Keep other variables that might affect the response the
same for all groups.
4. Replication. Use enough experimental units in each group so that
any differences in the effects of the treatments can be distinguished
from chance differences between the groups.
 Completely Randomized Design
In a completely randomized design, the treatments are assigned
to all the experimental units completely by chance.
Some experiments may include a control group that receives an
inactive treatment or an existing baseline treatment.

Treatment
Group 1 1

Experimental Compare
Units Random
Assignment Results

Treatment
Group 2
2
 Develop a flow chart for the caffeine experiment.

15 Drink Diet
students Coke

30 AP Stats Compare
Students Random
Assignment pulse rates

Drink
15 Caffeine
students free Diet
Coke
A food company wants to assess the nutrition quality of a new
“instant breakfast” by feeding it to thirty newly weaned male rats
over a 28 day period. A control group eats a standard diet but
otherwise eats the same as the experimental group.
Define variables and treatments:
Explanatory variables: Diet
Response variable: Nutritional Levels
Treatments: New instant breakfast or standard rat pellets
Use a flowchart design to present the essential parts of the
experiment
 Group 1
Std rat pellets
Sample Random 30 Compare
Nutritional
60 rats Allocation
levels
Group 2
Instant breakfast
30

Explain how each of the four principles of Experimental

Design was used.
 Blocking
When a population consists of groups of individuals that are “similar
within but different between,” a stratified random sample gives a better
estimate than a simple random sample. This same logic applies in
experiments.

A block is a group of experimental units that are known before the

experiment to be similar in some way that is expected to affect the
response to the treatments.
In a randomized block design, the random assignment of
experimental units to treatments is carried out separately within each
block.
 Akita

Beagle

Blocking Dogs
Collie Experiment/Simulation Dalmatian
 Experiments: What Can Go Wrong?
The logic of a randomized comparative experiment depends on
our ability to treat all the subjects the same in every way except
for the actual treatments being compared.

Good experiments, therefore, require careful attention to details

to ensure that all subjects really are treated identically.
The response to a dummy treatment is called the placebo effect.

In a double-blind experiment, neither the subjects nor those who

interact with them and measure the response variable know which
treatment a subject received.
 Inference for Experiments
In an experiment, researchers usually hope to see a difference in the
responses so large that it is unlikely to happen just because of chance
variation.

We can use the laws of probability, which describe chance behavior, to

learn whether the treatment effects are larger than we would expect to
see if only chance were operating.

If they are, we call them statistically significant.

An observed effect so large that it would rarely occur by chance is

called statistically significant.

A statistically significant association in data from a well-designed

experiment does imply causation.
 Matched Pairs Design
A common type of randomized block design for comparing two
treatments is a matched pairs design. The idea is to create blocks
by matching pairs of similar experimental units.

A matched pairs design is a randomized blocked experiment in which

each block consists of a matching pair of similar experimental units.

Chance is used to determine which unit in each pair gets each

treatment.

Sometimes, a “pair” in a matched-pairs design consists of a single unit

that receives both treatments. Since the order of the treatments can
influence the response, chance is used to determine with treatment is
applied first for each unit.
Matched Pairs Design
 Matched Pairs Design
• Matching the subjects in various ways can produce
more precise results than simple randomization
• Matched pairs design compares 2 treatments.
Subjects matched in pairs.
• Fitness example: Pair females with each other,
males with each other, one person in each pair
goes to one treatment group (weights), the other
person goes to the other treatment group (pilates)
• Before and after experiments: Try one diet for 6
months and get results, then try another diet
afterwards.
 You have recruited 10 right-handed people to serve
as subjects. Carefully describe the design of a
matched pairs experiment to compare the strength
of the right and left hands, using these subjects.
Squeeze Compare
Squeeze
Left hand
Right Combine
second strength
10 Right Random first (5) results
Handed Allocation and
Subjects Squeeze Squeeze Compare Compare
Left first Right hand hand
(5) second strength strength
 Review Game
• https://www.superteachertools.net/jeopardyx
/jeopardy-review-
game.php?gamefile=1401814749#.VFkdf2ewL
Mo