Types of Experiments &

Research Designs

UAPP 702: Research Design for Urban & Public Policy

Based on notes from Steven W. Peuquet. Ph.D.
 Types of Experiments
There are four kinds of experiments:
 True experiments
 Quasi-experiments
 Natural experiments
 Naturalistic experiments
 True Experiments

Some true experiments are done

in the lab, others are done in the
field.

There are five steps to follow in

conducting true experiments:
 True Experiments

1. Need at least two groups: an experimental

group and a control group. One groups
gets the intervention, the other group does
not.

2. Random assignment of individuals to the

groups. (The degree to which random
assignment ensures equivalence of the
groups is dependant upon the size of the
groups.)
 True Experiments
3. The groups are measured on one or more
dependent variables. This is called the
pretest.

4. The intervention (independent variable) is

introduced.

5. The dependent variables are measured

again. This is the post test.
 Quasi-Experiments

Are experiments where the ability to

randomly assign individuals to the
experimental and control groups is
limited or nonexistent.

Quasi-experiments are more commonly

used in evaluating social programs.
 Quasi-Experiments
Example
In a study of the effectiveness of a new
math curriculum, a 6th grade class in
Porter Middle School was not given the
new curriculum (the “control group”),
while the 6th grade class in Farley Middle
School was given the new curriculum
(experimental group).
 Quasi-Experiments
Example (cont.)

At the end of the school year the 6th

graders in the Farley MS class had higher
average math achievement test scores
than the kids in the Porter MS 6th grade
class.

Is this a “quasi-experiment”? Why?

 Quasi-Experiments
Example (cont.)
Answer: Most likely -- YES
Random assignment to the two classes
was probably not made, hence, groups
may not be equivalent. Possibility of
results being confounded by socio-
economic status (SES) or other factors.
 Natural Experiments
Natural experiments are happening
around us all the time. They are not
conducted by researchers, but simply
evaluated by researchers. In other words,
the researcher does not have control over
the application of the treatment. This also
means that there is no control over what
groups receive the treatment and the
composition of those groups.
 Natural Experiments
Examples:

Migration patterns of people — some people

migrate, some don’t.

Some middle class Latino students go to

college, some don’t.

Note: Each of these situations tests something

about human behavior, the question is: “What
is being tested by what is going on?”
 Naturalistic Experiments

In naturalistic experiments, one

contrives to collect experimental
data under natural conditions. You
make the data happen out in the
natural world (not in the lab), and
you evaluate the results.
 Naturalistic Experiments

The difference between a natural

experiment and a naturalistic
experiment is that the first just
happens, the second must be
contrived to happen.
 Naturalistic Experiments

Naturalistic experiments deviate

from true experiments because
group membership is not randomly
assigned, and exogenous factors
(confounding variables) are not
controlled.
 Naturalistic Experiments
Examples:

Links needed to connect any two randomly

selected people (passage of folders)

Patience of drivers behind a car that did not

move soon after a red light turned green
(independent variables: newness of car and
dress of driver)

Matched pair testing to uncover discrimination

 Internal and External
Validity of an Experiment
 Internal Validity
High internal validity means that
changes in the dependent variable
were caused by—not merely related to
or correlated with—the treatment.

High internal validity can be attained

only in an ideal true experimental
design, or when steps have been taken
to eliminate confounding factors or
influences.
 Internal Validity

Internal validity can be increased by:

Repeating an experiment many times, and by

changing or adding independent variables;

Switching the control and experimental

groups

Controlling confounding influences.

 Internal Validity

Controlled experiments have

the virtue of producing results
with high internal validity, but
they have the liability of low
external validity
 External Validity

External validity is the extent to

which the cause and effect
relationship can be generalized
to other groups that were not
part of the experiment.
 External Validity
External validity is critically
important in policy and program
evaluation research because we
would like our research findings to
be generalizable. If external
validity is low, then the results are
not generalizable.
 Threats to Validity

History confound —
Any independent variable other
than the treatment variable, that
occurs between the pre-test and
the post-test.
 Threats to Validity
Maturation confound —
Refers to the fact that as the
experiment is conducted, people
are growing older, and are
getting more experienced.
(Problem in longitudinal studies
in particular)
 Threats to Validity

Testing confound —

Refers to different responses

that an experimental subject
may have to questions they are
asked over and over again.
 Threats to Validity
Instrumentation confound —
Results from changing
measurement instruments.
(Example: use of different people
to collect data, if there is no inter-
rate reliability, you have changed
the measurement instrument.)
 Threats to Validity

Regression to the mean —

Groups tend to increasingly

resemble the mean over time.
 Threats to Validity

Selection bias confound —

When self-selection by
individuals into groups is
possible, a selection bias can
result.
 Threats to Validity

Diffusion of treatment
confound —

Occurs when a control group

cannot be prevented from
receiving the treatment in an
experiment.
 Examples

Let’s look at two related but

different research questions,
and see what types of
experiments would be
appropriate for each.
Final Important Principle!

NEVER USE A RESEARCH

DESIGN OF LESS LOGICAL
POWER WHEN ONE OF
GREATER LOGICAL POWER
IS FEASIBLE !