LESSON 5: QUASI-EXPERIMENTAL RESEARCH DESIGNS

Quasi-experimental designs came about because of:

1) difficulty of applying the classical natural science method to the social

sciences
2) overemphasis on theory testing and development
3) high cost of classic natural science methods
4) development of new statistical tools that allowed for statistical control
 
There are several types of quasi-experimental designs, including:
1) time series design
2) equivalent time series samples
3) equivalent samples materials design
4) non-equivalent control group
5) counterbalanced designs
6) separate sample pre-test/post-test
7) separate sample pre-test/post-test control group
8) multiple time series design
9) institutional cycle design
10) regression-discontinuity design

Comparison Group Pre-test/Post-test Design

In a quasi-experimental design, the research substitutes statistical

"controls" for the absence of physical control of the experimental situation.  The
most common quasi-experimental design is the Comparison Group Pre-
test/Post-test Design.  This design is the same as the classic controlled
experimental design except that the subjects cannot be randomly assigned to
either the experimental or the control group, or the researcher cannot control
which group will get the treatment.  In other words, participants do not all
have the same chance of being in the control or the experimental groups, or of
receiving or not receiving the treatment.

This design can be diagrammed as follows:

    O1    X    O2

    O1            O2

This can be illustrated by the following research study to determine whether a

home weatherization program for low income families reduced home energy
consumption.
 
 Average energy Average energy
Were homes Was home
consumption per home consumption per home
weatherized? weatherized?
(first measurement) (second measurement)
BTUs used per month; 
BTUs used per month; 
Home was BTUs per sf per degree
Yes BTUs per sf per degree
weatherized day; Cost per degree
day; Cost per degree
day
day
BTUs used per month; 
BTUs used per month; 
No (on the BTUs per sf per degree
BTUs per sf per degree
waiting list) day; Cost per degree
day; Cost per degree
day
day 

The weatherized homes saved an average of 10.95% of the BTUs used per
month before weatherization; saved an average of 1.68 BTUs per degree day per
square foot; and saved an average of $.006 per degree day.

The non-weatherized homes used more BTUs per month (+2.48% more)
over the same time period compared to the weatherized homes.  The non-
weatherized homes also used more BTUs per degree day per square foot; and
saved less money (an average of $.001 per degree day).

The difference between the change in the experimental group (down

10.95%) and the change in the control group (up 2.48%) is +13.43%.   This is
the amount of savings that can be attributed to the weatherization program.
The quasi-experimental design is not as strong in controlling for threats to the
internal and external validity of the study as the true controlled experimental
design.
 

Controlling for Threats to Internal Validity

1) History:  did some other current event effect the change in the
dependent variable?  No, because both groups experienced the same
current events.
2) Maturation:  were changes in the dependent variable due to normal
developmental processes?  No, because both groups experienced the
same developmental processes.

3) Statistical Regression:  did subjects come from low or high performing

groups?  Both groups were low income families but not necessarily high
energy users.

4) Selection:  were the subjects self-selected into experimental and control

groups, which could affect the dependent variable?  No, both groups had
 applied to the weatherization program, and had similar floor space,
number of occupants, and percent owner-occupied.

5) Experimental Mortality:  did some subjects drop out?  did this affect
the results?  There were some homes eliminated because of moves, being
away from home, or unable to get accurate fuel records.

6) Testing:  Did the pre-test affect the scores on the post-test?  No, both
groups supplied energy records.

7) Instrumentation:  Did the measurement method change during the

research?  No, both groups supplied energy records.

8) Design contamination:  did the control group find out about the
experimental treatment?  did either group have a reason to want to make
the research succeed or fail?  None noted.
 

Controlling for Threats to External Validity

1) Unique program features:  None noted.

2) Effects of Selection:  None noted.

3) Effects of Setting:  Study was done at one location in Minnesota;

4) Effects of History:  Study was done during a period of high energy costs;

5) Effects of Testing:  None noted.

6) Reactive effects of experimental arrangements:  Need to replicate the

findings in other locations and other time periods.

Quasi-experimental designs may be weak in controlling for threats to

internal validity, but can be quite strong in controlling for threats to external
validity.  It may be difficult to control which police are switched to a four-day
ten-hour shift, or which children are given a new method of learning a foreign
language.  However, since the research takes place in a natural setting, it may
have wide applicability to other similar settings.
 

Interrupted Time Series Design

This design uses several waves of observation before and after the
introduction of the independent (treatment) variable X.  It is diagrammed as
follows:

O1    O2    O3    O4    X    O5    O6    O7    O8

 This may be illustrated by a study designed to test whether the
implementation of a crackdown on speeding in a given state reduces the traffic
fatality rate in that state.
 

T-4 T-3 T-2 T-1 X T+1 T+2 T+3 T+4

Fatality  Fatality Fatality Fatality Fatality Fatality Fatality Fatality
Crack 
Rate  Rate  Rate  Rate  Rate  Rate  Rate  Rate 
(4 yrs  (3 yrs  (2 yrs (Year (Year (2 yrs (3 yrs (4 yrs
-down
before) before) before) before)  after ) after)  after) after)

This type of design works best if the treatment (independent variable) is

expected to have an immediate, marked effect, and if the treatment is
introduced (implemented) all at once in all relevant situations.  However, the
results may be difficult to interpret, especially if no statistically significant
differences are found.  Researchers must often collect qualitative data to
supplement and interpret the quantitative measurements.
 

The design is not particularly strong at controlling for threats to internal

validity:
1) History:  did some other current event effect the change in the
dependent variable?  Researcher must gather qualitative data on
possible events that could have affected the fatality rate.
2) Maturation:  were changes in the dependent variable due to
normal developmental processes?  No.
3) Statistical Regression:  did subjects come from low or high
performing groups?  Statistical analysis is used to determine
whether changes are due to statistical regression or the
independent variable.
4) Selection:  were the subjects self-selected into experimental and
control groups, which could affect the dependent variable? 
Researcher must determine whether there were any major changes
in the population between the before and after measures.
5) Experimental Mortality:  did some subjects drop out?  did this
affect the results?  Researcher must check whether some of the
population dropped out after the implementation of the crackdown.
6) Testing:  Did the pre-test affect the scores on the post-test?  No
effects.
7) Instrumentation:  Did the measurement method change during
the research?  Researcher must ensure that fatalities were
measured in the same way in all the years considered.
8) Design contamination:  did the control group find out about the
experimental treatment?  did either group have a reason to want to
make the research succeed or fail?  None noted.
Nor is this design strong on controlling for threats to external validity.

All the possible threats must be considered, in particular any interaction

between the selection of this population and the particular treatment
(crackdown) applied.  These concerns include a) unique program features; b)
effects of selection; c) effects of setting; d) effects of history; e) effects of testing;
f) reactive effects of experimental arrangements.

Interrupted Time Series Design with Comparison Group

The addition of a second time series for a comparison group helps to

provide a check on some of the threats to validity of the Single Interrupted
Time Series Design discussed above, especially history.

This design uses several waves of observation in both groups (treatment

and comparison groups) before and after the introduction of the independent
variable X in the treatment group.  It is diagrammed as follows:

State A:    O1    O2    O3    O4    X    O5    O6    O7    O8
State B:    O1    O2    O3    O4     -     O5    O6    O7    O8

This may be illustrated by a study to assess the effect of a crackdown on

drunk driving on automobile fatalities in one state, compared to automobile
fatalities in another state without a similar crackdown.
 

State  T-3 T-2 T-1 X T+1 T+2 T+3

Fatality  Fatality  Fatality  Crack  Fatality  Fatality  Fatality 
A
Rate Rate Rate down Rate Rate Rate
Fatality  Fatality  Fatality  Fatality  Fatality  Fatality 
B -
Rate Rate Rate Rate Rate Rate