Quasi-Experiments

The Basic Nonequivalent Groups Design

(NEGD)
N
N

O
O

O
O

Key Feature: Nonequivalent

assignment

What Does Nonequivalent Mean?

Assignment is nonrandom.
Researcher didnt control assignment.
Groups may be different.
Group differences may affect outcomes.

Equivalence

Equivalent groups are not necessarily

identical on any pre-test measure.
Merely implies that if the random
assignment procedure was repeated,
the groups would tend toward
equivalence.

Non-Equivalence

Non-equivalent groups do not necessarily

differ on any pre-test measure.
Merely implies that If the same nonrandom assignment procedure was
repeated, the groups would tend to toward
non-equivalence.
If assignment to groups was based partly
on income, then groups would tend to
have different expected mean levels of
income but any two groups you picked
might well be similar in income levels.

The Point

Equivalence or non-equivalence is
defined by the selection procedure.
Even if the difference in pre-test
means across groups is small,
this does not imply that the groups
are equivalent.
Small differences can introduce big threats.

Quasi- vs. Natural vs. Experiment

In a true experiment, the researcher performs

the random assignment
Can be in a lab or the field

In a natural experiment, someone else assigns

through a random process.
In a quasi-experiment, assignment is not
random, introducing selection threats.
Much stronger if the selection is not done by the
cases themselves (exogenous sorting).

What is a Natural Experiment

Strict Definition:
Some truly natural process, such as rainfall
or weather patterns, assigns IV.

Definition we all use in our own work:

Some exogenous process, rather than our
cases, ourselves, or a causal process
relevant to our theory, assigns IV.

Genres of Natural Experiments

The natural border or
natural disaster

The Rule Change

Jared Diamonds islands

Dan Posners rivers
Caroline Hoxbys streams
Settler mortality
(Acemoglu, Johnson, and
Robinson)
Hurricane Katrina

House seniority system

(Crooks and Hibbing)
GAVEL amendment in
Colorado
Connecticut speeding law
New Zealand electoral
reform
Propositions

Strength is that nature

doesnt care about your
cases or IV

Relatively easy to spot,

hard to defend

Genres of Natural Experiments

The Court Decision

Roe V. Wade for Levitt

and Donohue
Iowa item veto decision
Strength is that court is
not a blatant political
actor responding to
societal shifts or societal
pressures

The Lottery

James Fowlers use of

Canadian bill introduction
privilege
US House Clerk conducts a
randomization of the order in
which members choose
office
Strength is true randomness
in first step, but human
action in 2nd

Genres of Natural Experiments

Staged Implementation

Two-step
reapportionment
revolution in the United
States
Lots of program
evaluations in
development
Helps to rule out history
and maturation threats

The Threshold
Mail ballot assignment in
precincts with <250
voters

Need to make the

threshold unrelated to DV,
or else use Trochim-style
regression discontinuity

What Makes a Convincing Natural

Experiment?

You can show that the process of selection was

not related to characteristics of the cases that
are relevant to your DV
In a cross-sectional experiment, demonstrate
that the two groups are quite similar
In a time-series experiment, demonstrate that
little else changed when the treatment took
place.
In a word, show equivalence

Any purported causal test of needs to take

into consideration all of the two-group
threats to validity.
R
R

O
O

N
N

O
O

Can be a valid causal test.

Fully exposed to threats.

NEGD Design has Multiple Groups AND

Multiple Measures

N O X O
N O
O

This helps rule out

(or at least recognize)
threats.

Pre-Tests v. Covariates

N O X O
N O
O

Pre- Post-Test Design:

Observations are tests
you administer.

N O1 X O2
N O1
O2

Proxy Pre-Test Design:

First observations are
covariates on which you
collect data.

Problems of Internal Validity in

NEGDs

Internal Validity

N O X O
N O
O

All designs suffer from threats to validity.

In addition to all the single group threats,
quasi-experiments are particularly likely to suffer
from multi-group threats.

Selection-history
Selection-maturation
Selection-testing
Selection-instrumentation
Selection-regression
Selection-mortality

The Bivariate Distribution

90
80

Posttest

70
60
50
40
30
30

40

50

Pretest

60

70

80

The Bivariate Distribution

90
80

Posttest

70
60
50
40
30
30

40

60
70
80 a
Program
Group
has
pretest5-point pretest
advantage.

50

The Bivariate Distribution

90
80

Posttest

70

Program
group
scores
15-points
higher
on
Posttest.

60
50
40
30
30

40

Program
group
has
60
70
80 a
pretest5-point pretest
advantage,

50

Graph of Means
80
75
70
65
60
55
50
45
40
35
30

Comparison
Program

Pretest

Comp
Prog
ALL

pretest
MEAN
49.991
54.513
52.252

Posttest

posttest
MEAN
50.008
64.121
57.064

pretest
STD DEV
6.985
7.037
7.360

posttest
STD DEV
7.549
7.381
10.272

Possible Outcome #1
70
65
60
Comparison
Program

55
50
45
40
Pretest

Selection-history
Selection-maturation
Selection-testing
Selection-instrumentation
Selection-regression
Selection-mortality

Posttest

Possible: local event

Possible: PG initially higher
Unlikely: no change in CG
Possible: scale effects
Unlikely: expect change in CG
Possible: PG loses low scorers

Possible Outcome #2

Selection-history
Selection-maturation
Selection-testing
Selection-instrumentation
Selection-regression
Selection-mortality

Likely: PG initially higher

Likely: PG initially higher
Possible
Possible
Unlikely: expect change in CG
Possible: both lose low scorers

Possible Outcome #3
70
65
60
Comparison
Program

55
50
45
40
Pretest

Selection-history
Selection-maturation
Selection-testing
Selection-instrumentation
Selection-regression
Selection-mortality

Posttest

Possible: local event

Unlikely: no change in CG
Unlikely: no change in CG
Possible: scale effects
Likely
Possible: PG loses high
scorers

Possible Outcome #4
70
65
60
Comparison
Program

55
50
45
40
Pretest

Selection-history
Selection-maturation
Selection-testing
Selection-instrumentation
Selection-regression
Selection-mortality

Posttest

Possible: local event

Unlikely: no change in CG
Unlikely: no change in CG
Possible: scale effects
Very Likely
Possible: PG loses low scorers

Possible Outcome #5
70
65
60
Comparison
Program

55
50
45
40
Pretest

Posttest

Selection-history
Selection-maturation
Selection-testing And you should be so lucky
Selection-instrumentation
Selection-regression
Selection-mortality

Analysis Requirements
N
N

O
O

O
O

Pre-post (or covariates)

Two-group
Treatment-control (dummy = 0, 1)

Analysis of Covariance (ANCOVA)

where:
yi
0
1
2
Xi
Zi
ei

yi = 0 + 1Xi + 2Zi + ei

=
outcome score for the ith unit
=
coefficient for the intercept
=
pretest coefficient
=
mean difference for treatment
=
covariate
=
dummy variable for treatment(0 = control, 1=
treatment)
=
residual for the ith unit

The Bivariate Distribution

90
80

posttest

70

Program
group
scores
15-points
higher
on
Posttest.

60
50
40
30
30

40

Program
group
has
60
70
80 a
pretest5-point pretest
Advantage.

50

The Bivariate Distribution

90
80

posttest

70

Slope is
B1

Vertical
Distance is
Mean
Treatment
Effect, or
B2

60
50
40
30
30

40

50

pretest

60

70

80

Why Add Covariates to Analysis?

ANCOVA can include more than one

pretest or control variable.
Additional pretests further adjust for
initial group differences.
Ideally, in the absence of any treatment
effect, the covariates would perfectly
predict the posttest.
Additional covariates will often improve
the accuracy of the estimate of the
treatment effect.

Irrelevant Covariates

Adding pretests that are completely

unrelated to the posttest, however,
actually decreases precision.
Irrelevant covariates contribute
nothing to the analysis, but subtract a
degree of freedom from the error term.
This reduces the efficiency of the
estimate.

Omitted Covariates

Covariates that are related to the posttest but

not to the treatment can be ignored without
biasing the estimate of the treatment effect.
Covariates that are related to the posttest and
the treatment but that are omitted will bias the
estimate of the treatment effect.
We can safely omit control variables even if
they are highly correlated with the posttest as
long as they do not correlate with the
treatment.

Omitted Variables Bias

Omitted (relevant) covariates that are

positively correlated with the treatment
will lead us to overestimate the
treatment effect.
Omitted (relevant) covariates that are
negatively correlated with the treatment
will lead us to underestimate the
treatment effect.

Bottom Line

We should always try to include omitted

relevant covariates, except
When the omitted covariate is itself a
consequence of the treatment.
If cannot include a relevant covariate,
we can at least predict the direction if
not magnitude of the likely bias.

ButWhat about measurement error?

With multiple covariates, measurement

error does not always lead to a pseudoeffect.
As measurement error in any single
variable increases, it becomes as if
the variable is not included in the
ANCOVA.
This then mimics an omitted variables
problem, and the direction of bias
depends upon the relationship between
the noisy covariate and the treatment.

Other Quasi-Experimental Designs

Separate Pre-Post Samples

N1
N1
N2
N2

O
X

O
O

Groups with the same subscript come from the same

context.
Here, N1 might be people who were in the program at
Agency 1 last year, with those in N2 at Agency 2 last year.
This is like having a proxy pretest on a different group.

Separate Pre-Post Samples

N
N

R1
R1
R2
R2

O
X

O
O

Take random samples at two times of people at two

nonequivalent agencies.
Useful when you routinely measure with surveys.
You can assume that the pre and post samples are
equivalent, but the two agencies may not be.

Double-Pretest Design
N
N

O
O

O
O

O
O

Strong in internal validity

Helps address selection-maturation

Switching Replications
N
N

O
O

O
O

O
O

Strong design for both internal and

external validity
Strong against social threats to internal
validity
Strong ethically

Nonequivalent Dependent Variables Design (NEDV)

O1
O2

O1
O2

The variables have to be similar enough that

they are affected the same way by all threats.
The program has to target one variable and
not the other.
In simple form, weak internal validity.

NEDV Example

Only works if we can assume that geometry scores

show what would have happened to algebra if
untreated.
The variable is the control.
Note that there is no control group here.

NEDV Pattern Matching

Have many outcome variables.

Have theory that tells how affected
(from most to least) each variable will
be by the program.
Match observed gains with predicted
ones.
With pattern, NEDV can be extremely
powerful.

NEDV Pattern Matching

A ladder graph.

r = .997

NEDV: Lake and OMahony 2006

Hypothesis:
As territory
declines in
value in 20th
century
(measured by
average state
size), wars
fought over
territory should
decline in
frequency. There
should be no
pattern in other
Issues.