Professional Documents
Culture Documents
Donald Campbell PDF
Donald Campbell PDF
EXPERIMENTAL
QUASI-EXPERIMENTAL
DESIGNSFORGENERALIZED
CAUSALINFERENCE
ii:.
William R. Shadish
Trru UNIvERSITYop MEvPrrts
.jr-*"'"+.'-, ,
iLli"
fr
**
Thomas D. Cook
NonrrrwpsrERN UNrvPnslrY
Donald T. Campbell
HOUGHTONMIFFLINCOMPANY
and
Experiments
Causal
Generalized
lnference
Ex.per'i'ment (ik-spEr'e-mant):[Middle English from Old French from Latin
experimentum, from experiri, to try; seeper- in Indo-European Roots.]
n. Abbr. exp., expt, 1. a. A test under controlled conditions that is
made to demonstratea known truth, examine the validity of a hypothesis, or determine the efficacyof something previously untried' b. The
processof conducting such a test; experimentation. 2' An innovative
"Democracy is only an experiment in gouernment"
act or procedure:
(.V{illiam Ralph lnge).
Cause (k6z): [Middle English from Old French from Latin causa' teason,
purpose.] n. 1. a. The producer of an effect, result, or consequence.
b. The one, such as a person, an event' or a condition, that is responsible for an action or a result. v. 1. To be the causeof or reason for; result in. 2. To bring about or compel by authority or force.
o MANv historians and philosophers,the increasedemphasison experimentation in the 15th and L7th centuriesmarked the emergenceof modern science
1983). Drake (1981) cites
from its roots in natural philosophy (Hacking,
'Water,
'1.6'!.2
or Moue in It as usheringin
treatrseBodies Tbat Stay Atop
Galileo's
modern experimental science,but earlier claims can be made favoring \Tilliam
Gilbert's1,600study Onthe Loadstoneand MagneticBodies,Leonardoda Vinci's
and perhapseventhe Sth-centuryB.C.philoso(1,452-1.51.9)
many investigations,
pher Empedocles,who used various empirical demonstrationsto argue against
'1.969a,
1'969b).In the everyday senseof the term, humans
Parmenides(Jones,
have beenexperimentingwith different ways of doing things from the earliestmoments of their history. Suchexperimentingis as natural a part of our life as trying
a new recipe or a different way of starting campfires.
z | 1. EXeERTMENTs
ANDGENERALTzED
cAUsALINFERENcE
I
AND CAUSATTONI I
EXPERTMENTS
sciencein the laboratoryworked poorly in thesenew applications.So they developed new methodsof dealingwith extraneousinfluence,such as random assignment (Fisher,1,925)or addinga nonrandomizedcontrol group (Coover& Angell,
1.907).As theoreticaland observationalexperienceaccumulatedacrossthesesettings and topics,more sourcesof bias were identifiedand more methodswere developedto copewith them (Dehue,2000).
TodaSthe key featurecommonto all experimentsis still to deliberatelyvary
somethingso asto discoverwhat happensto somethingelselater-to discoverthe
what
effectsof presumedcauses.As laypersonswe do this, for example,to assess
happensto our blood pressureif we exercisemore, to our weight if we diet less,
or ro our behaviorif we read a self-helpbook. However,scientificexperimentation has developedincreasinglyspecializedsubstance,language,and tools, inthat is the pricluding the practiceof field experimentationin the socialsciences
mary focus of this book. This chapter begins to explore these matters by
(1) discussing
the natureof causationthat experimentstest,(2) explainingthe spethat decializedterminology(e.g.,randomizedexperiments,quasi-experiments)
generalize
problem
how
to
(3)
of
the
scribessocial experiments, introducing
causalconnectionsfrom individual experiments,and (4) briefly situatingthe experimentwithin a largerliteratureon the nature of science.
AND CAUSATION
EXPERIMENTS
A sensiblediscussionof experimentsrequiresboth a vocabularyfor talking about
causationand an understandingof key conceptsthat underliethat vocabulary.
DefiningCause,Effect,and CausalRelationships
Most peopleintuitively recognizecausalrelationshipsin their daily lives.For instance,you may say that another automobile'shitting yours was a causeof the
damageto your car; that the number of hours you spentstudyingwas a causeof
your testgrades;or that the amountof food a friend eatswas a causeof his weight.
You may evenpoint to more complicatedcausalrelationships,noting that a low
test gradewas demoralizing,which reducedsubsequentstudying,which caused
evenlower grades.Here the samevariable(low grade)can be both a causeand an
effect,and there can be a reciprocal relationship betweentwo variables (low
gradesand not studying)that causeeachother.
Despitethis intuitive familiarity with causalrelationsbips,a precisedefinition
of causeand effecthaseludedphilosophersfor centuries.lIndeed,the definitions
1. Our analysisrefldctsthe useof the word causationin ordinary language,not the more detaileddiscussionsof
causeby philosophers.Readersinterestedin suchdetail may consult a host of works that we referencein this
chapter,includingCook and Campbell(1979).
AND GENERALTZED
CAUSAL
INFERENCE
| 1. EXPERTMENTS
AND CAUSATIONI S
EXPERIMENTS
know which details are important, so it might take you some time to work it out"
(Rowe, L999, p.732). Endostatin was an inus condition. It was insufficientcause
by itself, and its effectivenessrequired it to be embedded in a larger set of conditions that were not even fully understood by the original investigators.
Most causesare more accurately called inus conditions. Many factors are usually required for an effectto occur, but we rarely know all of them and how they
relate to each other. This is one reason that the causal relationships we discussin
this book are not deterministic but only increasethe probability that an effect will
occur (Eells,1,991,;Holland, 1,994).It also explains why a given causalrelationship will occur under some conditions but not universally across time, space,hu-"r pop,rlations, or other kinds of treatments and outcomes that are more or less
related io those studied. To different {egrees, all causal relationships are context
dependent,so the generalizationof experimental effects is always at issue.That is
*hy *. return to such generahzationsthroughout this book.
Effect
'We
that
can better understand what an effect is through a counterfactual model'l'973'
goes back at least to the 18th-century philosopher David Hume (Lewis,
p. SSel. A counterfactual is something that is contrary to fact. In an experiment,
ie obseruewhat did happez when people received a treatment. The counterfactual is knowledge of what would haue happened to those same people if they simultaneously had not receivedtreatment. An effect is the difference betweenwhat
did happen and what would have happened.
'We
cannot actually observe a counterfactual. Consider phenylketonuria
metabolic diseasethat causesmental retardation unless
(PKU), a genetically-based
treated during the first few weeks of life. PKU is the absenceof an enzyme that
would otherwise prevent a buildup of phenylalanine, a substance toxic to the
nervous system. Vhen a restricted phenylalanine diet is begun early and maintained, reiardation is prevented. In this example, the causecould be thought of as
the underlying genetic defect, as the enzymatic disorder, or as the diet. Each implies a difierenicounterfactual. For example, if we say that a restricted phenylalanine diet causeda decreasein PKU-basedmental retardation in infants who are
at birth, the counterfactual is whatever would have happened
phenylketonuric
'h"d
t'h.r. sameinfants not receiveda restricted phenylalanine diet. The samelogic
applies to the genetic or enzymatic version of the cause. But it is impossible for
theseu.ry ,"-i infants simultaneously to both have and not have the diet, the genetic disorder, or the enzyme deficiency.
So a central task for all cause-probing research is to create reasonable approximations to this physically impossible counterfactual. For instance, if it were
ethical to do so, we might contrast phenylketonuric infants who were given the
diet with other phenylketonuric infants who wer not given the diet but who were
similar in many ways to those who were (e.g., similar face) gender,age, socioeconomic status, health status). Or we might (if it were ethical) contrast infants who
6 I 1. EXPERIMENTS
ANDGENERALIZED
CAUSAL
INFERENCE
were not on the diet for the first 3 months of their lives with those same infants
after they were put on the diet starting in the 4th month. Neither of these approximations is a true counterfactual. In the first case,the individual infants in the
treatment condition are different from those in the comparison condition; in the
second case, the identities are the same, but time has passedand many changes
other than the treatment have occurred to the infants (including permanent damage done by phenylalanine during the first 3 months of life). So two central tasks
in experimental design are creating a high-quality but necessarilyimperfect source
of counterfactual inference and understanding how this source differs from the
treatment condition.
This counterfactual reasoning is fundarnentally qualitative becausecausal inference, even in experiments, is fundamentally qualitative (Campbell, 1975;
Shadish, 1995a; Shadish 6c Cook, 1,999). However, some of these points have
been formalized by statisticiansinto a specialcasethat is sometimescalled Rubin's
"1.974,'1.977,1978,79861.
CausalModel (Holland, 1,986;Rubin,
This book is not
about statistics, so we do not describethat model in detail ('West,Biesanz,& Pitts
[2000] do so and relate it to the Campbell tradition). A primary emphasisof Rubin's model is the analysis of causein experiments, and its basic premisesare consistent with those of this book.2 Rubin's model has also been widely used to analyze causal inference in case-control studies in public health and medicine
(Holland 6c Rubin, 1988), in path analysisin sociology (Holland,1986), and in
a paradox that Lord (1967) introduced into psychology (Holland 6c Rubin,
1983); and it has generatedmany statistical innovations that we cover later in this
book. It is new enough that critiques of it are just now beginning to appear (e.g.,
Dawid, 2000; Pearl, 2000). tUfhat is clear, however, is that Rubin's is a very general model with obvious and subtle implications. Both it and the critiques of it are
required material for advanced students and scholars of cause-probingmethods.
CausalRelationship
How do we know if cause and effect are related? In a classic analysis formalized
by the 19th-century philosopher John Stuart Mill, a causal relationship exists if
(1) the causeprecededthe effect, (2) the causewas related to the effect,and (3) we
can find no plausible alternative explanation for the effect other than the cause.
These three characteristics mirror what happens in experiments in which (1) we
manipulate the presumed cause and observe an outcome afterward; (2) we see
whether variation in the cause is related to variation in the effect; and (3) we use
various methods during the experiment to reduce the plausibility of other explanations for the effect, along with ancillary methods to explore the plausibility of
those we cannot rule out (most of this book is about methods for doing this).
2. However, Rubin's model is not intended to say much about the matters of causal generalization that we address
in this book.
EXPERTMENTS
AND CAUSATTON
| 7
I
and Confounds
Correlation,
Causation,
A well-known maxim in research is: Correlation does not proue causation. This is
so becausewe may not know which variable came first nor whether alternative explanations for the presumed effectexist. For example, supposeincome and education are correlated.Do you have to have a high income before you can aff.ordto pay
for education,or do you first have to get a good education before you can get a better paying job? Each possibility may be true, and so both need investigation.But until those investigationsare completed and evaluatedby the scholarly communiry a
simple correlation doesnot indicate which variable came first. Correlations also do
little to rule out alternative explanations for a relationship between two variables
such as education and income. That relationship may not be causal at all but rather
due to a third variable (often called a confound), such as intelligence or family socioeconomicstatus,that causesboth high education and high income. For example,
if high intelligencecausessuccessin education and on the job, then intelligent people would have correlatededucation and incomes,not becauseeducation causesincome (or vice versa) but becauseboth would be causedby intelligence.Thus a central task in the study of experiments is identifying the different kinds of confounds
that can operate in a particular researcharea and understanding the strengthsand
weaknessesassociatedwith various ways of dealing with them
Causes
and Nonmanipulable
Manipulable
In the intuitive understandingof experimentationthat most peoplehave,it makes
senseto say,"Let's seewhat happensif we requirewelfarerecipientsto work"; but
it makesno senseto say,"Let's seewhat happensif I changethis adult maleinto a
Experimentsexplore
girl." And so it is alsoin scientificexperiments.
three-year-old
of a medicine,the
the
dose
such
as
the effectsof things that can be manipulated,
amount of a welfarecheck,the kind or amount of psychotherapyor the number
of childrenin a classroom.Nonmanipulableevents(e.g.,the explosionof a supernova) or attributes(e.g.,people'sages,their raw geneticmaterial,or their biologiwe cannotdeliberatelyvary them
cal sex)cannotbe causesin experimentsbecause
most scientistsand philosophersagree
to seewhat then happens.Consequently,
that it is much harderto discoverthe effectsof nonmanipulablecauses.
1. EXeERTMENTS
ANDGENERALTzED
cAUsALTNFERENcE
EXPERIMENTS
AND CAUSATIONI 9
and CausalExplanation
CausalDescription
attribThe uniquestrengthof experimentationis in describingthe consequences
utableto deliberatelyvaryinga treatment.'Wecall this causaldescription.In contrast, experimentsdo lesswell in clarifying the mechanismsthrough which and
the conditionsunder which that causalrelationshipholds-what we call causal
explanation.For example,most childrenvery quickly learnthe descriptivecausal
relationshipbetweenflicking a light switch and obtainingillumination in a room.
However,few children (or evenadults)can fully explain why that light goeson.
To do so, they would haveto decomposethe treatment(the act of flicking a light
switch)into its causallyefficaciousfeatures(e.g.,closingan insulatedcircuit) and
its nonessentialfeatures(e.g.,whetherthe switch is thrown by hand or a motion
detector).They would haveto do the samefor the effect (eitherincandescentor
fluorescentlight can be produced,but light will still be produced whether the
light fixture is recessedor not). For full explanation,they would then have to
show how the causallyefficaciousparts of the treatmentinfluencethe causally
affectedparts of the outcomethrough identified mediating processes(e.g.,the
INFERENCE
ANDGENERALIZED
CAUSAL
1O I T. CXPTRIMENTS
passageof electricity through the circuit, the excitation of photons).3 ClearlS the
causeof the light going on is a complex cluster of many factors. For those philosophers who equate cause with identifying that constellation of variables that necessarily inevitably and infallibly results in the effect (Beauchamp,1.974),talk of
cause is not warranted until everything of relevanceis known. For them, there is
no causal description without causal explanation. Whatever the philosophic merits of their position, though, it is not practical to expect much current social science to achieve such complete explanation.
The practical importance of causal explanation is brought home when the
switch fails to make the light go on and when replacing the light bulb (another
easily learned manipulation) fails to solva the problem. Explanatory knowledge
then offers clues about how to fix the problem-for example, by detecting and repairing a short circuit. Or if we wanted to create illumination in a place without
lights and we had explanatory knowledge, we would know exactly which features
of the cause-and-effectrelationship are essentialto create light and which are irrelevant. Our explanation might tell us that there must be a source of electricity
but that that source could take several different molar forms, such as abattery, a
generator, a windmill, or a solar array. There must also be a switch mechanism to
close a circuit, but this could also take many forms, including the touching of two
bare wires or even a motion detector that trips the switch when someone enters
the room. So causal explanation is an important route to the generalization of
causal descriptions becauseit tells us which features of the causal relationship are
essentialto transfer to other situations.
This benefit of causal explanation helps elucidate its priority and prestige in
all sciencesand helps explain why, once a novel and important causal relationship
is discovered, the bulk of basic scientific effort turns toward explaining why and
how it happens. Usuallg this involves decomposing the causeinto its causally effective parts, decomposing the effects into its causally affected parts, and identifying the processesthrough which the effective causal parts influence the causally
affected outcome parts.
These examplesalso show the close parallel between descriptive and explanatory causation and molar and molecular causation.aDescriptive causation usually
concerns simple bivariate relationships between molar treatments and molar outcomes, molar here referring to a package that consistsof many different parts. For
instance, we may find that psychotherapy decreasesdepression,a simple descriptive causal relationship benveen a molar treatment package and a molar outcome.
However, psychotherapy consists of such parts as verbal interactions, placebo3. However, the full explanationa physicistwould offer might be quite different from this electrician's
explanation,perhapsinvoking the behaviorof subparticles.This differenceindicatesiust how complicatedis the
notion of explanationand how it can quickly becomequite complex once one shifts levelsof analysis.
4. By molar, we mean somethingtaken as a whole rather than in parts. An analogyis to physics,in which molar
might refer to the propertiesor motions of masses,as distinguishedfrom those of moleculesor atomsthat make up
thosemasses.
EXPERIMENTS
AND CAUSATIONI 11
I
12 I 1. EXPERTMENTS
AND GENERALTZED
TNFERENCE
CAUSAL
I
MODERNDESCRIPTIONS
OF EXPERIMENTS
Some of the terms used in describing modern experimentation (seeTable L.L) are
unique, clearly defined, and consistently used; others are blurred and inconsistently used. The common attribute in all experiments is control of treatment
(though control can take many different forms). So Mosteller (1990, p. 225)
writes, "fn an experiment the investigator controls the application of the treatment"l and Yaremko, Harari, Harrison, and Lynn (1,986,p.72) write, "one or
more independent variables are manipulated to observe their effects on one or
more dependentvariables." However, over time many different experimental subtypes have developed in responseto the needs and histories of different sciences
'Winston
('Winston, 1990;
6c Blais, 1.996\.
TABLE1.1TheVocabularyof Experiments
Experiment:
A studyin whichan intervention
to observe
itseffects.
is deliberately
introduced
Randomized
Experiment:
to receive
the treatmentor
An experiment
in whichunitsareassigned
an alternative
conditionby a randomprocess
suchasthe tossof a coinor a tableof
randomnumbers.
randomly.
An experiment
in whichunitsarenot assigned
to conditions
Quasi-Experiment:
NaturalExperiment:
Not reallyan experiment
because
the causeusuallycannotbe
manipulated;
with
a studythat contrasts
eventsuchasan earthquake
occurring
a naturally
a comoarison
condition.
Correlational
or observational
study;a study
Study:Usuallysynonymous
with nonexperimental
thatsimplyobserves
amongvariables.
the sizeanddirection
of a relationship
OF EXPERIMENTS
MODERNDESCRIPTIONS
I tr
Experiment
Randomized
The most clearlydescribedvariant is the randomizedexperiment,widely credited
to Sir RonaldFisher(1,925,1926).Itwas first usedin agriculturebut laterspread
to other topic areasbecauseit promisedcontrol over extraneoussourcesof variation without requiringthe physicalisolationof the laboratory.Its distinguishing
featureis clear and important-that the varioustreatmentsbeingcontrasted(includingno treatmentat all) are assignedto experimentalunits' by chance,for example,by cointossor useof a table of random numbers.If implementedcorrectlS
,"rdo- assignmentcreatestwo or more groupsof units that are probabilistically
Hence,any outcomedifferencesthat are obsimilarto .".h other on the average.6
servedbetweenthosegroupsat the end,ofa study arelikely to be dueto treatment'
not to differencesbetweenthe groupsthat alreadyexistedat the start of the study.
Further,when certainassumptionsare met, the randomizedexperimentyieldsan
estimateof the sizeof a treatmenteffectthat has desirablestatisticalproperties'
along with estimatesof the probability that the true effectfalls within a defined
confidenceinterval.Thesefeaturesof experimentsare so highly prized that in a
researchareasuchas medicinethe randomizedexperimentis often referredto as
the gold standardfor treatmentoutcomeresearch.'
Closelyrelatedto the randomizedexperimentis a more ambiguousand inconsistentlyusedterm, true experiment.Someauthorsuseit synonymouslywith
randomizedexperiment(Rosenthal& Rosnow,1991').Others useit more genermanipally to refer to any studyin which an independentvariableis deliberately
'We
ulated (Yaremkoet al., 1,9861anda dependentvariableis assessed. shall not
usethe term at all givenits ambiguity and given that the modifier true seemsto
imply restrictedclaimsto a singlecorrectexperimentalmethod.
Quasi-Experiment
Much of this book focuseson a class of designsthat Campbell and Stanley
sharewith all other
(1,963)popularizedasquasi-experiments.s
Quasi-experiments
5. Units can be people,animals,time periods,institutions,or almost anything else.Typically in field
experimentationthey are peopleor someaggregateof people,such as classroomsor work sites.In addition, a little
thought showsthat random assignmentof units to treatmentsis the sameas assignmentof treatmentsto units, so
thesephrasesare frequendyusedinterchangeably'
6. The word probabilisticallyis crucial, as is explainedin more detail in Chapter 8.
7. Although the rerm randomized experiment is used this way consistently acrossmany fields and in this book,
statisticianssometimesuse the closely related term random experiment in a different way to indicate experiments
for which the outcomecannor be predictedwith certainry(e.g.,Hogg & Tanis, 1988).
8. Campbell (1957) first calledthesecompromisedesignsbut changedterminologyvery quickly; Rosenbaum
(1995a\ and Cochran (1965\ referto theseas observationalstudies,a term we avoid becausemany peopleuseit to
to
refer to correlationalor nonexperimentalstudies,as well. Greenbergand Shroder(1997) usequdsi-etcperiment
refer to studiesthat randomly assigngroups (e.g.,communities)to conditions,but we would considerthesegrouprandomizedexperiments(Murray' 1998).
14 I 1. EXPERIMENTS
AND GENERALIZED
CAUSAL
INFERENCE
I
experiments a similar purpose-to test descriptivecausal hypothesesabout manipulable causes-as well as many structural details, such as the frequent presenceof
control groups and pretest measures,to support a counterfactual inference about
what would have happened in the absenceof treatment. But, by definition, quasiexperiments lack random assignment. Assignment to conditions is by means of selfselection,by which units choosetreatment for themselves,or by meansof administrator selection,by which teachers,bureaucrats,legislators,therapists,physicians,
or others decide which persons should get which treatment. Howeveq researchers
who use quasi-experimentsmay still have considerablecontrol over selectingand
schedulingmeasures,over how nonrandom assignmentis executed,over the kinds
of comparison groups with which treatment,groups are compared, and over some
aspectsof how treatment is scheduled.As Campbell and Stanleynote:
There are many natural socialsettingsin which the researchpersoncan introduce
somethinglike experimentaldesigninto his schedulingof data collectionprocedures
(e.g.,the uhen and to whom of measurement),
eventhough he lacksthe full control
over the schedulingof experimentalstimuli (the when and to wltom of exposureand
the ability to randomizeexposures)which makesa true experimentpossible.Collecdesigns.(Campbell&
tively,such situationscan be regardedas quasi-experimental
p. 34)
StanleS1,963,
In quasi-experiments,the causeis manipulable and occurs before the effect is
measured. However, quasi-experimental design features usually create less compelling support for counterfactual inferences. For example, quasi-experimental
control groups may differ from the treatment condition in many systematic(nonrandom) ways other than the presenceof the treatment Many of theseways could
be alternative explanations for the observed effect, and so researchershave to
worry about ruling them out in order to get a more valid estimate of the treatment
effect. By contrast, with random assignmentthe researcherdoes not have to think
as much about all these alternative explanations. If correctly done, random assignment makes most of the alternatives less likely as causes of the observed
treatment effect at the start of the study.
In quasi-experiments,the researcherhas to enumeratealternative explanations
one by one, decide which are plausible, and then use logic, design, and measurement to assesswhether each one is operating in a way that might explain any observedeffect. The difficulties are that thesealternative explanations are never completely enumerable in advance, that some of them are particular to the context
being studied, and that the methods neededto eliminate them from contention will
vary from alternative to alternative and from study to study. For example, suppose
two nonrandomly formed groups of children are studied, a volunteer treatment
group that gets a new reading program and a control group of nonvolunteerswho
do not get it. If the treatment group does better, is it becauseof treatment or becausethe cognitive development of the volunteerswas increasingmore rapidly even
before treatment began? (In a randomized experiment, maturation rates would
rl
OF EXPERIMENTS 1s
MODERNDESCRIPTIONS
|
INFERENCE
AND GENERALIZED
CAUSAL
16 I 1. EXPERIMENTS
I
OFEXPERIMENTS
MODERNDESCRTPTONS
II tt
NaturalExperiment
The term natural experiment describesa naturally-occurring contrast between a
treatment and a comparisoncondition (Fagan, 1990; Meyer, 1995;Zeisel,1,973l.
Often the treatments are not even potentially manipulable, as when researchers
retrospectivelyexamined whether earthquakesin California causeddrops in property values (Brunette, 1.995; Murdoch, Singh, 6c Thayer, 1993). Yet plausible
causal inferences about the effects of earthquakes are easy to construct and defend. After all, the earthquakesoccurred before the observations on property values,and it is easyto seewhether earthquakesare related to properfy values. A useful source of counterfactual inference can be constructed by examining property
values in the same locale before the earthquake or by studying similar localesthat
did not experience an earthquake during the bame time. If property values
dropped right after the earthquake in the earthquake condition but not in the comparison condition, it is difficult to find an alternative explanation for that drop.
Natural experiments have recently gained a high profile in economics. Before
the 1990s economists had great faith in their ability to produce valid causal inferencesthrough statistical adjustments for initial nonequivalence between treatment and control groups. But two studies on the effects of job training programs
showed that those adjustments produced estimates that were not close to those
generated from a randomized experiment and were unstable across tests of the
model's sensitivity (Fraker 6c Maynard, 1,987; Lalonde, 1986). Hence, in their
searchfor alternative methods, many economistscame to do natural experiments,
such as the economic study of the effects that occurred in the Miami job market
when many prisoners were releasedfrom Cuban jails and allowed to come to the
United States(Card, 1990). They assumethat the releaseof prisoners (or the timing of an earthquake) is independent of the ongoing processesthat usually affect
unemployment rates (or housing values). Later we explore the validity of this
assumption-of its desirability there can be little question.
18 I 1. EXPERIMENTS
AND GENERALIZED
INFERENCE
CAUSAL
Nonexperimental
Designs
The termscorrelationaldesign,passiveobservationaldesign,and nonexperimental
designrefer to situationsin which a presumedcauseand effect are identified and
measuredbut in which other structural featuresof experimentsare missing.Random assignmentis not part of the design,nor are suchdesignelementsas pretests
and control groupsfrom which researchers
might constructa usefulcounterfactual
inference.Instead,relianceis placedon measuringalternativeexplanationsindividually and then statisticallycontrolling for them. In cross-sectional
studiesin
which all the data aregatheredon the respondentsat one time, the researchermay
not even know if the causeprecedesthe dffect. When thesestudiesare used for
causalpurposes,the missingdesignfeaturescan be problematicunlessmuch is already known about which alternativeinterpretationsare plausible,unlessthose
that are plausiblecan be validly measured,and unlessthe substantivemodel used
for statisticaladjustmentis well-specified.
Theseare difficult conditionsto meetin
the real world of researchpractice,and thereforemany commentatorsdoubt the
potentialof suchdesignsto supportstrongcausalinferencesin most cases.
EXPERIMENTS
ANDTHEGENERALIZATION
OF
CAUSALCONNECTIONS
The strength of experimentation is its ability to illuminate causal inference. The
weaknessof experimentation is doubt about the extent to which that causal rela'We
tionship generalizes.
hope that an innovative feature of this book is its focus
on generalization. Here we introduce the general issuesthat are expanded in later
chapters.
Most Experiments
Are HighlyLocalBut Have
GeneralAspirations
Most experimentsare highly localizedand particularistic.They are almostalways
conductedin a restrictedrange of settings,often just one, with a particular version of one type of treatmentrather than, say,a sampleof all possibleversions.
Usually they have severalmeasures-eachwith theoreticalassumptionsthat are
differentfrom thosepresentin other measures-but far from a completesetof all
possiblemeasures.Each experimentnearly always usesa convenientsampleof
people rather than one that reflectsa well-describedpopulation; and it will inevitably be conductedat a particular point in time that rapidly becomeshistory.
Yet readersof experimentalresultsare rarelyconcernedwith what happened
in that particular,past,local study.Rather,they usuallyaim to learn eitherabout
theoreticalconstructsof interestor about alarger policy.Theoristsoften want to
CONNECTIONS
OFCAUSAL
AND THEGENERALIZATION
EXeERTMENTS
I t'
INFERENCE
20 I 1. EXPERIMENTS
AND GENERALIZED
CAUSAL
ConstructValidity:CausalGeneralization
as Representation
The first causal generalization problem concerns how to go from the particular
units, treatments, observations, and settings on which data are collected to the
higher order constructs these instancesrepresent.These constructs are almost always couched in terms that are more abstract than the particular instancessampled in an experiment. The labels may pertain to the individual elementsof the experiment (e.g., is the outcome measured by a given test best described as
intelligence or as achievement?).Or the labels may pertain to the nature of relationships among elements, including causal relationships, as when cancer treatments are classified as cytotoxic or cytostatic depending on whether they kill tumor cells directly or delay tumor growth by modulating their environment.
Consider a randomized experiment by Fortin and Kirouac (1.9761.The treatment
was a brief educational course administered by severalnurses,who gave a tour of
their hospital and covered some basic facts about surgery with individuals who
were to have elective abdominal or thoracic surgery 1-5to 20 days later in a single Montreal hospital. Ten specific outcome measureswere used after the surgery,
such as an activities of daily living scaleand a count of the analgesicsused to control pain. Now compare this study with its likely t^rget constructs-whether
10. \Weoccasionallyrefer to time as a separatefeatureof experiments,following Campbell (79571and Cook and
Campbell (19791,becausetime can cut acrossthe other factorsindependently.Cronbachdid not includetime in
his notational system,insteadincorporating time into treatment(e.g.,the schedulingof treatment),observations
(e.g.,when measuresare administered),or setting (e.g.,the historicalcontext of the experiment).
oF cAUsALcoNNEcrtoNS| ,,
ANDTHEGENERALIZATIoN
EXnERTMENTs
I
patient education (the target cause)promotes physical recovery (the targt effect)
"*ong surgical patients (the target population of units) in hospitals (the target
univeise ofiettings). Another example occurs in basic research,in which the question frequently aiises as to whether the actual manipulations and measuresused
in an experiment really tap into the specific cause and effect constructs specified
by the theory. One way to dismiss an empirical challenge to a theory is simply to
make the casethat the data do not really represent the concepts as they are specified in the theory.
Empirical resnlts often force researchersto change their initial understanding
of whaithe domain under study is. Sometimesthe reconceptuahzation leads to a
more restricted inference about what has been studied. Thus the planned causal
agent in the Fortin and Kirouac (I976),study-patie,nt education-might need to
b! respecified as informational patient education if the information component of
the treatment proved to be causally related to recovery from surgery but the tour
of the hospital did not. Conversely data can sometimes lead researchersto think
in terms o?,"rg., constructs and categoriesthat are more general than those with
which they began a researchprogram. Thus the creative analyst of patient education studies mlght surmise that the treatment is a subclass of interventions that
"perceived control" or that recovery from surgery can be
function by increasing
;'p.tronal coping." Subsequentreaders of the study can
treated as a subclas of
even add their own interpietations, perhaps claiming that perceived control is really just a special caseof the even more general self-efficacy construct. There is a
sobtie interplay over time among the original categories the researcherintended
to represeni, the study as it was actually conducted, the study results, and subseqrr..ri interpretations. This interplay can change the researcher'sthinking about
what the siudy particulars actually achieved at a more conceptual level, as can
feedback fromreaders. But whatever reconceptualizationsoccur' the first problem
of causal generaltzationis always the same: How can we generalizefrom a sample of instancesand the data patterns associatedwith them to the particular target constructs they represent?
as Extrapolation
ExternalValidity:CausalGeneralization
The secondproblem of generalizationis to infer whether a causalrelationship
holdsovervariationsin p.rrorrt, settings,treatments,and outcomes.For example,
someonereadingthe resultsof an experimenton the effectsof a kindergarten
grammarschoolreadingtestscoresof poor
Head Startprogiam on the subsequent
African Americanchildrenin Memphis during the 1980smay want to know if a
programwith partially overlappingcognitiveand socialdevelopmentgoals_would
be aseffectivein improvingthi mathematicstest scoresof poor Hispanicchildren
in Dallas if this programwere to be implementedtomorrow.
This exampl. again reminds us that generahzationis not a synonym for
broader applicatiorr.H.r., generahzationis from one city to another city and
1. EXPERIMENTS
AND GENERALIZED
INFERENCE
CAUSAL
from one kind of clienteleto anotherkind, but thereis no presumptionthat Dallas is somehow broader than Memphis or that Hispanic children constitute a
broader population than African American children. Of course,some generalizations are from narrow to broad. For example,a researcherwho randomly
samplesexperimentalparticipants from a national population may generalize
(probabilistically)from the sampleto all the other unstudiedmembersof that
samepopulation. Indeed,that is the rationale for choosingrandom selectionin
the first place.Similarly when policymakersconsiderwhetherHead Start should
be continuedon a national basis,they are not so interestedin what happenedin
Memphis.They are more interestedin what would happenon the averageacross
the United States,as its many local programsstill differ from eachother despite
efforts in the 1990sto standardizemuch of what happensto Head Startchildren
and parents.But generalizationcan also go from the broad to the narrow. Cronbetween
bach(1982)givesthe exampleof an experimentthat studieddifferences
the performancesof groups of studentsattendingprivate and public schools.In
this case,the concernof individual parentsis to know which type of schoolis better for their particular child, not for the whole group. \Thether from narrow to
broad, broad to narroq or acrossunits at about the samelevelof aggregation,
all theseexamplesof externalvalidity questionssharethe sameneed-to infer the
extent to which the effect holds over variationsin persons,settings,treatments,
or outcomes.
Approaches
to MakingCausalGeneralizations
\Thichever way the causal generalization issue is framed, experiments do not
seem at first glance to be very useful. Almost invariablS a given experiment uses
a limited set of operations to represent units, treatments, outcomes, and settings.
This high degree of localization is not unique to the experiment; it also characterizes case studies, performance monitoring systems, and opportunisticallyadministered marketing questionnaires given to, say, a haphazard sample of respondents at local shopping centers (Shadish, 1995b). Even when questionnaires
are administered to nationally representative samples, they are ideal for representing that particular population of persons but have little relevanceto citizens
outside of that nation. Moreover, responsesmay also vary by the setting in which
the interview took place (a doorstep, a living room, or a work site), by the time
of day at which it was administered, by how each question was framed, or by the
particular race, age,and gender combination of interviewers. But the fact that the
experiment is not alone in its vulnerability to generalization issuesdoes not make
it any less a problem. So what is it that justifies any belief that an experiment can
achieve a better fit between the sampling particulars of a study and more general
inferences to constructs or over variations in persons, settings, treatments, and
outcomes?
oF cAUsALcoNNEcrtoNs I tt
ANDTHEGENERALtzATtoN
EXeERTMENTs
Samplingand CausalGeneralization
The methodmost often recommendedfor achievingthis closefit is the useof formal probabiliry samplingof instancesof units, treatments,observations,or setthat we have clearly
tings (Rossi,Vlright, & Anderson,L983). This presupposes
deiineatedpopulationsof eachand that we can samplewith known probability
from within eachof thesepopulations.In effect,this entailsthe random selection
earof instances,to be carefullydistinguishedfrom random assignmentdiscussed
repreto
chance
by
lier in this chapter.Randomselectioninvolvesselectingcases
sentthat popuiation,whereasrandom assignmentinvolvesassigningcasesto multiple conditions.
In cause-probingresearchthat is not experimental,random samplesof indilongitudinalsurveyssuchasthe PanelStudyof
viduals"r. oft.n nr.d. Large-scale
IncomeDynamicsor the National Longitudinal Surveyare usedto representthe
populationof the United States-or certainagebracketswithin it-and measures
Lf pot.ntial causesand effectsare then relatedto each other using time lags in
All this is donein
,nr^"r.rr.-ent and statisticalcontrolsfor group nonequivalence.
hopesof approximatingwhat a randomizedexperimentachieves.However,cases
of random ielection from a broad population followed by random assignment
from within this population are much rarer (seeChapter 12 for examples).Also
Such
rare arestudiesoi t".rdotn selectionfollowed by a quality quasi-experiment.
that
control
logistical
experimentsrequirea high levelof resourcesand a degreeof
prefer to rely on an implicit set of nonstais iarely feasible,so many researchers
tistical heuristicsfor generalizationthat we hope to make more explicit and systematicin this book.
Random selectionoccurseven more rarely with treatments'outcomes,and
settingsthan with people.Considerthe outcomesobservedin an experiment.How
ofterrlre they raniomly sampled?'Wegrant that the domain samplingmodel of
classicaltestiheory (Nunnally 6c Bernstein,1994)assumesthat the itemsusedto
measurea constructhavebeenrandomly sampledfrom a domain of all possible
items. However,in actual experimentalpracticefew researchersever randomly
sampleitemswhen constructingmeasures.Nor do they do so when choosingmanipulationsor settings.For instance,many settingswill not agreeto be sampled,
"rid ,o1n. of the settingsthat agreeto be randomly sampledwill almostcertainly
not agreeto be randomlyassignedto conditions.For treatments,no definitivelist
of poisible treatmentsusuallyexists,as is most obvious in areasin which treatare being discoveredand developedrapidly, such as in AIDS research.In
-*,,
general,then, random samplingis alwaysdesirable,but it is only rarely and confeasible.
tingently
"However,
formal samplingmethodsare not the only option. Two informal, purposive samplingmethodrare sometimesuseful-purposive sampling of heterogeneousinstancesand purposivesamplingof typical instances.In the former case'the
aim is to includeinrLni.r chosendeliberatelyto reflect diversity on presumptively
important dimensions,eventhough the sampleis not formally random. In the latter
INFERENCE
CAUSAL
ANDGENERALIZED
24 I .l. TxpEnIMENTS
case,the aim is to explicate the kinds of units, treatments, observations, and settings
to which one most wants to generalize andthen to selectat least one instance of each
class that is impressionistically similar to the class mode. Although these purposive
sampling methods are more practical than formal probability sampling, they are not
backed by a statistical logic that justifies formal generalizations.Nonetheless, they
are probabty the most commonly used of all sampling methods for facilitating generalizations. A task we set ourselvesin this book is to explicate such methods and to
describe how they can be used more often than is the casetoday.
However, sampling methods of any kind are insufficient to solve either problem of generalization. Formal probability sampling requires specifying a target
population from which sampling then takes place, but defining such populations
is difficult for some targets of generalization such as treatments. Purposive sampling of heterogeneousinstancesis differentially feasible for different elementsin
a study; it is often more feasible to make measuresdiverse than it is to obtain diverse settings, for example. Purposive sampling of typical instancesis often feasible when target modes, medians, or means are known, but it leaves questions
about generalizationsto a wider range than is typical. Besides,as Cronbach points
out, most challenges to the causal generalization of an experiment typically
emerge after a study is done. In such cases,sampling is relevant only if the instancesin the original study were sampled diversely enough to promote responsible reanalysesof the data to seeif a treatment effect holds acrossmost or all of the
targets about which generahzation has been challenged. But packing so many
sourcesof variation into a single experimental study is rarely practical and will almost certainly conflict with other goals of the experiment. Formal sampling methods usually offer only a limited solution to causal generalizationproblems. A theory of generalizedcausal inference needsadditional tools.
A GroundedTheoryof CausalGeneralization
in their research,and
Practicingscientistsroutinely make causal generalizations
do. In this book, we
they
they almostneveruseformal probability samplingwhen
that is groundedin the actualpracticeof
presenta theory of causalgeneralization
science(Matt, Cook, 6c Shadish,2000). Although this theory was originally developedfrom ideasthat were groundedin the constructand externalvalidiry lithavesincefound that theseideasarecommonin
eratures(Cook, 1990,1991.),we
(e.g.,Abelson,1995;Campbell
a diverseliteratureabout scientificgeneralizations
& Fiske, 1.959;Cronbach& Meehl, 1955; Davis, 1994; Locke, 1'986;Medin,
Hayward,Tu1,991';'$7ilson,
1989;Messick,1ggg,1'995;Rubins,1.994;'Willner,
grounded
theory
this
about
\7e providemore details
nis, Bass,& Guyatt, 1,995];t.
that scientistsmakecausalgenin Chapters1L through L3, but in brief it suggests
eralizationsin their work by usingfive closelyrelatedprinciples:
"L.
the apparentsimilaritiesbetweenstudy operaSurfaceSimilarity.They assess
of the target of generalization.
tions and the prototypicalcharacteristics
OFCAUSALCONNECTIONS
AND THEGENERALIZATION
EXPERIMENTS
II ZS
2. Ruling Out lrreleuancies.They identify those things that are irrelevant because
they do not change a generalization.
Discriminations. They clarify k.y discriminations that limit
Making
3.
generalization.
4. Interpolation and Extrapolation. They make interpolations to unsampled values within the range of the sampled instances and, much more difficult, they
explore extrapolations beyond the sampled range.
5 . Causal Explanation. They develop and test explanatory theories about the pattern of effects,causes,and mediational processesthat are essentialto the transfer of a causalrelationship.
In this book, we want to show how scientistscan and do use thesefive principles to draw generalizedconclusions dbout a causal connection. Sometimes the
conclusion is about the higher order constructs to use in describing an obtained
connection at the samplelevel. In this sense,thesefive principles have analoguesor
parallels both in the construct validity literature (e.g.,with construct content, with
loru.rg.nt and discriminant validity, and with the need for theoretical rationales
for consrructs) and in the cognitive scienceand philosophy literatures that study
how people decidewhether instancesfall into a category(e.g.,concerning the roles
that protorypical characteristicsand surface versus deep similarity play in determining category membership). But at other times, the conclusion about generalization refers to whether a connection holds broadly or narrowly over variations
in persons, settings,treatments, or outcomes. Here, too, the principles have analogues or parallels that we can recognizefrom scientific theory and practice, as in
the study of dose-responserelationships (a form of interpolation-extrapolation) or
the appeal to explanatory mechanismsin generalizing from animals to humans (a
form of causal explanation).
Scientistsuse rhese five principles almost constantly during all phases of research.For example, when they read a published study and wonder if some variathink about similarition on the study's particulars would work in their lab, they
'$7hen
they conceptualize
ties of the published study to what they propose to do.
plan
will match the
to
study
they
instances
the new study, they anticipate how the
prototypical featuresof the constructs about which they are curious. They may deiign their study on the assumptionthat certain variations will be irrelevant to it but
that others will point to key discriminations over which the causal relationship
does not hold or the very character of the constructs changes.They may include
measuresof key theoretical mechanisms to clarify how the intervention works.
During data analysis, they test all these hypotheses and adjust their construct descriptions to match better what the data suggest happened in the study. The introduction section of their articles tries to convince the reader that the study bears on
specific constructs, and the discussion sometimes speculatesabout how results
-igttt extrapolate to different units, treatments, outcomes, and settings.
Further, practicing scientistsdo all this not just with single studies that they
read or conduct but also with multiple studies. They nearly always think about
26
INFERENCE
CAUSAL
1. EXPERTMENTS
ANDGENERALIZED
how their own studiesfit into a larger literature about both the constructsbeing
measuredand the variablesthat may or may not bound or explain a causalconnection, often documentingthis fit in the introduction to their study.And they apply all
five principleswhen they conduct reviewsof the literature,in which they make inthat a body of researchcan suppoft.
ferencesabout the kinds of generalizations
Throughoutthis book, and especiallyin Chapters11 to L3, we providemore
detailsabout this groundedtheory of causal generalizationand about the scientific
doesnot
Adopting this groundedtheoryof generalization
practicesthat it suggests.
we recommendsuchsamimply a rejectionof formal probabilitysampling.Indeed,
to
purposive
samplingschemes
pling unambiguouslywhenit is feasible,alongwith
when formal randomselectionmethodscannotbe implemented.
aid generalization
But we alsoshow that samplingis just one methodthat practicingscientistsuseto
along with practicallogic, applicationof diversestamake causalgeneralizations,
tistical methods,and useof featuresof designother than sampling.
AND METASCIENCE
EXPERIMENTS
Extensivephilosophicaldebatesometimessurroundsexperimentation.Here we
briefly summarizesomekey featuresof thesedebates,and then we discusssome
implications of thesedebatesfor experimentation.However,there is a sensein
which all this philosophicaldebateis incidentalto the practiceof experimentation.
Experimentationis as old as humanity itself, so it precededhumanity'sphilosophicaleffortsto understandcausationand genenlizationby thousandsof years.
Even over just the past 400 yearsof scientificexperimentation,we can seesome
constancyof experimentalconcept and method, whereasdiversephilosophical
"Exof the experimenthavecomeand gone.As Hacking(1983)said,
conceptions
most
perimentationhas a life of its own" (p. 150). It has beenone of science's
powerful methodsfor discoveringdescriptivecausalrelationships,and it hasdone
so well in so many ways that its placein scienceis probably assuredforever.To
justify its practicetodag a scientistneednot resortto sophisticatedphilosophical
reasoningabout experimentation.
Nonetheless,it doeshelp scientiststo understandthesephilosophicaldebates.
For example,previousdistinctionsin this chapterbetweenmolar and molecular
causation,descriptiveand explanatorycause,or probabilisticand deterministic
causalinferencesall help both philosophersand scientiststo understandbetter
both the purposeand the resultsof experiments(e.g.,Bunge,1959; Eells, 1991';
Hart & Honore, 1985;Humphreys,"t989;Mackie, 1'974;Salmon,7984,1989;
Sobel,1993;P.A. \X/hite,1990).Here we focus on a differentand broadersetof
critiquesof scienceitself,not only from philosophybut alsofrom the history,sociologS and psychologyof science(seeusefulgeneralreviewsby Bechtel,1988;
H. I. Brown, 1977; Oldroyd, 19861.Someof theseworks have beenexplicitly
about the nature of experimentation,seekingto createa justified role for it (e.g.,
AND METASCIENCE
EXPERIMENTS
I 27
'1.990;
S. Drake, l98l; Gergen,
Bhaskar,L975;Campbell,1982,,1988;Danziger,
Pinch,6cSchaffer,
L989;
Gooding,
Houts,
Neimeyer,6d
1,973;Gholson,Shadish,
'Woolgar,
1,989b;Greenwood, L989; Hacking, L983; Latour, 1'987;Latour 6c
& Fuller,L994;
1988;Orne,1.962;R.RosenthaL,1.966;Shadish
1.979;Morawski,
Thesecritiqueshelp scientiststo seesomelimits of experimentaShapin,1,9941.
tion in both scienceand society.
TheKuhnianCritique
Kuhn (1962\ describedscientificrevolutionsas differentand partly incommensueachother in time and in which the gradrableparadigmsthat abruptly succeedgd
of scientificknowledgewas a chimera.Hanson(1958),Polanyi
ual accumulation
Toulmin (1'961),Feyerabend(L975),and Quine (1'95t'
(1958),Popper('J.959),
1,969)contributedto the critical momentum,in part by exposingthe grossmistakesin logicalpositivism'sattemptto build a philosophyof sciencebasedon reconstructinga successfulsciencesuch as physics.All thesecritiquesdeniedany
firm foundationsfor scientificknowledge(so, by extension,experimentsdo not
provide firm causalknowledge).The logicalpositivistshopedto achievefoundations on which to build knowledgeby tying all theory tightly to theory-freeobservationthrough predicatelogic. But this left out important scientificconcepts
that could not be tied tightly to observation;and it failed to recognizethat all observationsare impregnatedwith substantiveand methodologicaltheory,making
it impossibleto conducttheory-freetests.lt
The impossibility of theory-neutral observation (often referred to as the
Quine-Duhemthesis)impliesthat the resultsof any singletest (and so any single
experiment)are inevitably ambiguous.They could be disputed,for example,on
groundsthat the theoreticalassumptionsbuilt into the outcome measurewere
wrong or that the study made a fatity assumptionabout how high a treatment
dosewas requiredto be effective.Someof theseassumptionsare small,easilydetected,and correctable,suchaswhen a voltmetergivesthe wrong readingbecause
the impedanceof the voltagesourcewas much higherthan that of the meter ('$filson, L952).But other assumptionsare more paradigmlike,impregnatinga theory
so completelythat other parts of the theory makeno sensewithout them (e.g.,the
assumptionthat the earthis the centerof the universein pre-Galileanastronomy).
Becausethe number of assumptionsinvolved in any scientifictest is very large,
researcherscan easily find some assumptionsto fault or can even posit new
28
INFERENCE
CAUSAL
ANDGENERALIZED
r. rxeenlMENTs
Critiques
ModernSocialPsychological
working within traditionsvariouslycalledsocialconstructivism,episSociologists
temologicalrelativism,and the strongprogram(e.g.,Barnes,1974;Bloor,1976;
1'979)have
Collins,l98l;Knorr-Cetina,L981-;Latour 6c'Woolgar,1.979;Mulkay,
at work in science.Their empiricalstudies
shown thoseextrascientificprocesses
show that scientistsoften fail to adhereto norms commonlyproposedas part of
good science(e.g.,objectivity neutrality,sharingof information).They havealso
rho*n how that which comesto be reportedas scientificknowledgeis partly determinedby socialand psychologicalforcesand partly by issuesof economicand
political power both within scienceand in the largersociety-issuesthat arerarely
mention;d in publishedresearchreports.The most extremeamongthesesocioloclaiming
gistsattributesall scientificknowledgeto suchextrascientificprocesses,
"the
natural world has a small or nonexistentrole in the constructionof sciihat
"l'98I,
p. 3).
entificknowledge"(Collins,
Collins doesnot denyontologicalrea.lism,that real entitiesexistin the world.
Rather,he deniesepistemological(scientific)realism, that whateverexternal reality may existcanconstrainour scientifictheories.For example,if atomsreally exist, do they affectour scientifictheoriesat all? If our theory postulatesan atom, is
relit describing a realentitythat existsroughly aswe describeit? Epistetnologi,cal
atiuistssuch as Collins respondnegativelyto both questions,believingthat the
most important influencesin scienceare social,psychological,economic,and political, "ttd th"t thesemight evenbe the only influenceson scientifictheories-This
view is not widely endorsedoutsidea small group of sociologists,but it is a useful counterweightto naiveassumptionsthat scientificstudiessomehowdirectlyreveal natur. to r.r,(an assumptiorwe callnaiuerealism).The resultsof all studies,
including experiments,are profoundly subjectto theseextrascientificinfluences,
from their conceptionto reportsof their results.
and Trust
Science
A standard image of the scientist is as a skeptic, a person who only trusts results that
have been personally verified. Indeed, the scientific revolution of the'l'7th century
AND METASCIENCE
EXPERIMENTS
I 29
I
claimed that trust, particularly trust in authority and dogma, was antithetical to
good science.Every authoritative assertion,every dogma, was to be open to question, and the job of sciencewas to do that questioning.
That image is partly wrong. Any single scientific study is an exercisein trust
(Pinch, 1986; Shapin, 1,994).Studies trust the vast majority of already developed
methods, findings, and concepts that they use when they test a new hypothesis.
For example, statistical theories and methods are usually taken on faith rather
than personally verified, as are measurement instruments. The ratio of trust to
skepticism in any given study is more llke 99% trust to 1% skepticism than the
opposite. Even in lifelong programs of research, the single scientist trusts much
-or. than he or she ever doubts. Indeed, thoroughgoing skepticism is probably
impossible for the individual scientist, po iudge from what we know of the psychology of science(Gholson et al., L989; Shadish 6c Fuller, 1'9941.Finall5 skepticism is not even an accuratecharacterrzation of past scientific revolutions; Shapin
"gentlemanly trust" in L7th-century England was
(1,994) shows that the role of
central to the establishment of experimental science.Trust pervades science,despite its rhetoric of skepticism.
for Experiments
lmplications
The net result of thesecriticismsis a greaterappreciationfor the equivocalityof
all scientificknowledge.The experimentis not a clearwindow that revealsnature
directly to us.To the contrary,experimentsyield hypotheticaland fallible knowledgethat is often dependenton context and imbuedwith many unstatedtheoretical assumprions.Consequentlyexperimentalresultsare partly relativeto those
assumptionsand contextsand might well changewith new assumptionsor conconstructivistsand relativists.
texts.In this sense,all scientistsare epistemological
The differenceis whether they are strong or weak relativists.Strong relativists
share Collins'sposition that only extrascientificfactors influenceour theories.
'Weak
relativistsbelievethat both the ontologicalworld and the worlds of ideolog5 interests,values,hopes,and wishesplay a role in the constructionof scientiiic knowledge.Most practicingscientists,including ourselves,would probably
", Lrrtologicalrealistsbut weak epistemologicalrelativists.l2
describethemselves
To the extent that experimentsrevealnature to us, it is through a very clouded
windowpane(Campbell,1988).
to naiveviewsof experimentswere badly needed.As reSuchcounterweights
centlyas 30 yearsago,the centralrole of the experimentin sciencewas probably
1.2. If spacepermitred,we could exrendthis discussionto a host of other philosophicalissuesthat have beenraised
about the experiment, such as its role in discovery versusconfirmation, incorrect assertionsthat the experiment is
tied to somespecificphilosophysuch as logical positivismor pragmatism,and the various mistakesthat are
frequentlymadei., suchdiscussions(e.g.,Campbell, 1982,1988; Cook, 1991; Cook 6< Campbell, 1985; Shadish,
1.995a\.
I
INFERENCE
AND GENERALTZED
CAUSAL
30 | 1. EXPERTMENTS
taken more for granted than is the case today. For example, Campbell and Stan-
ley (1.9631
describedthemselvesas:
committed to the experiment: as the only means for settling disputes regarding educational practice, as the only way of verifying educational improvements, and as the only
way of establishing a cumulative tradition in which improvements can be introduced
without the danger of a faddish discard of old wisdom in favor of inferior novelties. (p. 2)
OR CAUSES?I gT
A WORLDWITHOUTEXPERIMENTS
I
ORCAUSES?
A WORLDWITHOUTEXPERIMENTS
To borrow a thought experimentfrom Maclntyre (1981),imaginethat the slates
of scienceand philosophywerewiped cleanand that we had to constructour understandingof the world anew.As part of that reconstruction,would we reinvent
the notion of a manipulablecause?\7e think so, largely becauseof the practical
utility that dependablemanipulandahave for our ability to surviveand prosper.
IUTouldwe reinvent the experimentas a method for investigatingsuch causes?
I
AND GENERALTZED
32 | 1. EXPERTMENTS
CAUSAL
TNFERENCE
A CriticalAssessment
of
Our Assumptions
As.sump.tion(e-simp'shen):[Middle Englishassumpcion,from Latin assumpti, assumptin-adoption, from assumptus,past participle of assmere,te adopt; seeassume.]n. 1. The act of taking to or upon oneself:
assumptionof an obligation. 2.The act of taking overiassumptionof
command. 3. The act of taking for granted:assumptionof a false theory. 4. Somethingtaken for granted or acceptedas true without proof;
a supposition:a ualid assumption. 5. Presumption;arrogance.5.
Logic.A minor premise.
fltHIS BooK covers five central topics across its 13 chapters. The first topic
| (Chapter 1) deals with our general understanding of descriptive causation and
I experimentation. The second (Chapters 2 and 3) deals with the types of validity and the specific validity threats associatedwith this understanding. The third
(Chapters 4 through 7) deals with quasi-experimentsand illustrates how combining design features can facilitate better causal inference. The fourth (Chapters 8
through L0) concerns randomized experiments and stressesthe factors that impede and promote their implementation. The fifth (Chapters 11 through L3) deals
with causal generalization, both theoretically and as concerns the conduct of individual studies and programs of research.The purpose of this last chapter is to
critically assesssome of the assumptions that have gone into these five topics, especially the assumptions that critics have found obiectionable or that we antici'We
pate they will find objectionable.
organize the discussionaround each of the
five topics and then briefly justify why we did not deal more extensivelywith nonexperimental methods for assessingcausation.
I7e do not delude ourselvesthat we can be the best explicators of our own assumptions. Our critics can do that task better. But we want to be as comprehensrve and
sive
an(l as explicit
explclt as we can.
can. This
I nrs is
part because
ls in
rn part
becausewe
we are
are convinced
convrnced of
ot the
the adaclvantages of falsification as a major component of any epistemology for the social
sciences,and forcing out one's assumptions and confronting them is one part of
falsification. But it is also becausewe would like to stimulate critical debateabout
theseassumptionsso that we can learn from those who would challengeour think456
rct
AND EXPERIMENTATION
CAUSATION
|
ing. If therewereto be a future book that carriedevenfurther forward the tradition emanatingfrom Campbelland Stanleyvia Cook and Campbellto this book,
then that futuie book *o,rld probably be all the better for building upon all the
justifiedcriticismscomingfrom thosewho do not agreewith us, eitheron particcau,rlu6 o, on the whole approachwe havetaken to the analysisof descriptive
would like this chapternot only to model the atsationand its generayzition.'We
but
i.-p, to be cr"iti.alabout the assumprionsall scholarsmust inevitablymake
be
might
they
how
and
alsoto encourageothersto think about theseassumptions
in fuiure empiricalor theoreticalwork'
addressed
ENTATION
AND EXPERIM
CAUSATION
CausalArrows and Pretzels
descriptive
Experiments test the influence of one or at most a small subset of
very few
causes.If statistical interactions are involved, they tend to be among
variables'
moderator
of
treatments or between a single treatment and a limited set
typical
Many researchersbelieve that the causal knowledge that results from this
structure fails to map the many causal forces that simultaneously af.*p.ii-..rtal
(e.g., Cronbach et al',
fe.t "ny given outcome in compiex and nonlinear ways
prioritize on ar19g0; Magnusson,2000). These critics assertthat experiments
an explanatory
,o*, .onrr-.cting A to B when they should instead seekto describe
most causal
pretzel or set of intersectingpretzels,as it were. They also believethat
whether
ielationships vary across ,rttitt, settings, and times, and so they doubt
Snow,
6c
(e.g.,
Cronbach
there ".. "ny constant bivariate causal relationships
reflect sta1977).Those that do appearto be dependablein the data may simply
reveal the
to
tistically underpow.r.i irr,, of modeiators or mediators that failed
sizesmight
true underlying complex causal relationships. True-variation in effect
or the
also be obrc.rr"d b.c"rrs. the relevant substantive theory is underspecified,
or
attenuated,
is
outcome measuresare partially invalid, or the treatment contrast
(McClelland
causally implicated variables afe truncated in how they are sampled
6c Judd, 1993).
As valid as theseobiectionsare, they do not invalidatethe casefor experiments.The purposeof experimentsis not to completelyexplain-some.phenomemakes
non; it is to ldentify whethera particularvariableor small setof variables
affecta margirraldifferencein someoutcomeover and above all the other forces
not
ing that outcome.Moreover,ontologicaldoubts such as the precedinghave
many
though
as
stJppedbelieversin more complex iausal theoriesfrom acting
or as
.r,rol relationshipscan be usefullycharacterizedas dependablemain effects
In this
very simpl. nonlin."rities that are also dependableenoughto be_u_seful.
where
connection,considersomeexamplesfrom educationin the United States,
4s8
14.A CRTTTCAL
ASSESSMENT
OFOURASSUMPTTONS
objections to experimentation are probably the most prevalent and virulent. Few
educational researchersseemto object to the following substantiveconclusions of
the form that A dependably causesB: small schools are better than large ones;
time-on-task raises achievement; summer school raises test scores;school desegregation hardly affects achievement but does increaseWhite flight; and assigning
and grading homework raises achievement.The critics also do not seemto object
to other conclusions involving very simple causal contingencies: reducing class
"sizable"
size increasesachievement,but only if the amount of change is
and to a
level under 20; or Catholic schools are superior to public ones, but only in the inner city and not in the suburbs and then most noticeably in graduation rates rather
,
than in achievementtest scores.
The primary iustification for such oversimplifications-and for the use of the
experiments that test them-is that some moderators of effects are of minor relevance to policy and theory even if they marginally improve explanation. The most
important contingencies are usually those that modify the sign of a causal relationship rather than its magnitude. Sign changesimply that a treatment is beneficial in some circumstancesbut might be harmful in others. This is quite different
from identifying circumstancesthat influence just how positive an effect might be.
Policy-makers are often willing to advocate an overall change,even if they suspect
it has different-sizedpositive effects for different groups, as long as the effects are
rarely negative. But if some groups will be positively affected and others negatively political actors are loath to prescribe different treatments for different
groups becauserivalries and jealousies often ensue. Theoreticians also probably
pay more attention to causal relationships that differ in causal sign becausethis
result implies that one can identify the boundary conditions that impel such a disparate data pattern.
Of course, we do not advocate ignoring all causal contingencies.For example, physicians routinely prescribe one of severalpossibleinterventions for a given
diagnosis.The exact choice may depend on the diagnosis,test results,patient preferences, insurance resources, and the availability of treatments in the patient's
area. However, the costs of such a contingent system are high. In part to limit the
number of relevant contingencies,physicians specialize,andwithin their own specialty they undergo extensivetraining to enable them to make thesecontingent decisions. Even then, substantial judgment is still required to cover the many situations in which causal contingencies are ambiguous or in dispute. In many other
policy domains it would also be costly to implement the financial, management,
and cultural changesthat a truly contingent system would require even if the requisite knowledge were available. Taking such a contingent approach to its logical
extremes would entail in education, for example, that individual tutoring become
matched
the order of the day.
dav.Studentsand instructorswould haveto be carefullymatched
AND EXPERIMENTATION
CAU5ATION
I Ot'
of E4periments
Criticisms
Epistemological
we have
In highlightingstatisticalconclusionvalidity and in-selectingexamples,
testing'
often linked causaldescriptionto quantitativemethodsand hypothesis
positivism'
Many criticswill (wrongly)r.. this asimplying a discreditedtheory of
positivismreAs a philosophyof scieniefirst outlined in the early L9th century'
about unobservables,and equated
1.ct.d' metaphysicalspeculations,especially
school of
lrro*t.ag. *lih descriptionsof e*periencedphenomena-A narrower
realism
logical pisitivism .*.rg.d in the eatly 20th century that also rejected
form
logic
*til. "lro .-phasizing Ih. ,rr. of data-theoryconnectionsin predicate
Both thesere""J " fr.f.r.r.. for p"redictingphenomenaover explainingthem'
*.r. lonf ago discredited,especiallyas explanationsof how
lated epistemologies
this basis'Howscienceop.r"trr.*so few criticsseriouslycritici'e experimentson
to attack
ever,many critics use the term positiuismwith lesshistorical fidelity
1985)'
quantitativesocialsciencemethodsin genera-l(e'g', Lincoln & Guba,
quantification
of
use
liuilding on the rejectionof logicalpositivism,they reiectthe
measurement,and hypothesistesting.Because
and forLal logic in observatiron,
of positheselast featuresare part of experiments,to reiectthis looseconception
are nutivism entailsrejectingexperiments.However,the errorsin suchcriticisms
(like the idea that
merous.For example,to ,eject a specificfeatureof positivism
only permissiblelinks betweendata and
f,r"rrtifi.rtion and p redicatelogicare the
imlly reiectingall relatedand more generalpropositiheory;doesnot nJcessarily
testing
tions jsuch asthe notion that somekinds of quantificationand hypothesis
ersuch
more
outlined
may be usefulfor knowledgegrowth).Ife and othershave
(Phillips,1990;Shadish,I995al'
rors elsewhere
criticismsof experimentationcitethe work of historians
other epistemological
and'woolof sciencesuchasKuh"n(1,g62),ofsociologistsof sciencesuchasLatour
tend
gar ltiZll "rrd of fhiloroph.ir of scienceiuchas Harr6'(1931).Thesecritics
that
notion
of theories,the
to focuson threethings.orre.i, the incommensurability
specifiedand so can alwaysbe reinterpreted.As a retheoriesare neverper"fectly
be reiected'its
sult, when disconfirmingdata seemto imply that a theory should
poriolut., can insteadbI reworkedin order to make the theory and observations
to the
consistentwith eachother.This is usuallydoneby addingnew contingencies
theory that limit the conditions under which it is thought to hold. A second critique is of the assumption that experimental observations can be used as truth
'We
tests.
would like observations to be objective assessmentsthat can adjudicate
between different theoretical explanations of a phenomenon. But in practice, observationsare not theory neutral; they are open to multiple interpretations that include such irrelevanciesas the researcher'shopes, dreams, and predilections. The
consequenceis that observations rarely result in definitive hypothesistests.The final criticism follows from the many behavioral and cognitive inconsistenciesbetween what scientists do in practice and what scientific norms prescribe they
should do. Descriptions of scientists' behavior in laboratories reveal them as
choosing to do particular experiments becausethey have an intuition about a relationship, or they are simply curious to seewhat happens, or they want to play
with a new piece of equipment they happen to find lying around. Their impetus,
therefore, is not a hypothesis carefully deduced from a theory that they then test
by means of careful observation.
Although these critiques have some credibilitg they are overgeneralized.Few
experimentersbelievethat their work yields definitive results even after it has been
subjected to professional review. Further, though these philosophical, historical,
and social critiques complicate what a "fact" means for any scientific method,
nonethelessmany relationships have stubbornly recurred despite changesassociated with the substantive theories, methods, and researcherbiasesthat first generated them. Observations may never achieve the status of "facts," but many of
them are so stubbornly replicable that they may be consideredas though they were
facts. For experimenters, the trick is to make sure that observations are not impregnated with just one theory, and this is done by building multiple theories into
observationsand by valuing independent replications, especiallythose of substantive critics-what we have elsewherecalled critical multiplism (Cook, 1985;
Shadish,'1.989,
1994).
Although causal claims can never be definitively tested and proven, individual experiments still manage to probe such claims. For example, if a study produces negative results, it is often the casethat program developersand other advocates then bring up methodological and substantive contingenciesthat might
have changedthe result. For instance, they might contend that a different outcome
measure or population would have led to a different conclusion. Subsequentstudies then probe these alternatives and, if they again prove negative, lead to yet another round of probes of whatever new explanatory possibilities have emerged.
After a time, this process runs out of steam, so particularistic are the contingencies that remain to be examined. It is as though a consensusemerges:"The causal
relationship was not obtained under many conditions. The conditions that remain
to be examined are so circumscribed that the intervention will not be worth much
'W'e
even if it is effectiveunder these conditions. "
agreethat this processis as much
or more social than logical. But the reality of elastic theory does not mean that decisions about causal hypotheses are only social and devoid of all empirical and
logical content.
I
I
t
AND EXPERIMENTATION
CAUSATION
| +er
NeglectedAncillarYQuestions
Our focus on causalquestionswithin an experimentalframework neglectsmany
other questionsthat arerelevantto causation.Theseincludequestionsabout how
to decideon the importanceor leverageof any singlecausalquestion.This could
entail exploringwhethera causalquestionis evenwarranted,as it often is not at
the early sa"g.-ofdevelopmentof an issue.Or it could entail exploringwhat type
of c".rsalquestionis moie important-one that fills an identifiedhole in someliterature,o, orr. that setsout to identify specificboundary conditionslimiting a
causalconnection,or one that probesthe validity of a centralassumptionheld by
within a field, or one that reducesuncertainty
all the theoristsand researchers
about an important decisionwhen formerly uncertaintywas high. Our approach
alsoneglectsthe realitythat how oneformulatesa descriptivecausalquestionusuaily enLils meetingsomestakeholders'interestsin the socialresearchmore than
those of others.TLus to ask about the effectsof a national program meetsthe
staffs,the media,and policy wonks to learnaboutwhether
needsof Congressional
the program"*orks. But it can fail to meet the needsof local practitionerswho
of microelementswithin the pro,rro"lly"*"nt to know about the effectiveness
gram ,o thut they can usethis knowledgeto improve their daily practice.-Inmore
Ih.or.ti."l work, to ask how some interventionaffectspersonalself-efficacyis
likely to promote individuals'autonomyneeds,whereasto ask about the effects
of a'persoasivecommunicationdesignedto changeattitudescould well cater to
462
14.A CR|T|CAL
ASSESSMENT
OFOURASSUMPT|ONS
the needs of those who would limit or manipulate such autonomy. Our narrow
technical approach to causation also neglectedissuesrelated to how such causal
knowledge might be used and misused. It gave short shrift to a systematic analysis of the kinds of causal questions that can and cannot be answered through experiments. \7hat about the effects of abortion, divorce, stable cohabitation, birth
out of wedlock, and other possibly harmful events that we cannot ethically manipulate? What about the effects of class,race, and gender that are not amenable
'What
to experimentation?
about the effects of historical occurrencesthat can be
studied only by using time-seriesmethods on whatever variables might or might
not be in the archives?Of what use, one might ask, is a method that cannot get at
some of the most important phenomena that shape our social world, often over
generations,as in the caseof race, class,and gender?
Many statisticians now consider questions about things that cannot be manipulated as being beyond causal analysis,so closely do they link manipulation to
causation. To them, the cause must be at least potentially manipulable, even if it
is not actually manipulated in a given observational study. Thus they would not
consider race ^ cause, though they would speak of the causal analysis of race in
studies in which Black and White couples are, say, randomly assignedto visiting
rental units in order to seeif the refusal rates vary, or that entail chemically changing skin color to seehow individuals are responded to differently as a function of
pigmentation, or that systematicallyvaried the racial mix of studentsin schools or
classrooms in order to study teacher responsesand student performance. Many
critics do not like so tight a coupling of manipulation and causation. For example, those who do status attainment researchconsider it obvious that race causally
influences how teachers treat individual minority students and thus affects how
well these children do in school and therefore what jobs they get and what
prospects their own children will subsequentlyhave. So this coupling of causeto
manipulation is a real limit of an experimental approach to causation. Although
we like the coupling of causation and manipulation for purposes of defining experiments, we do not seeit as necessaryto all useful forms of cause.
VALIDITY
Objectionsto InternalValidity
There are severalcriticismsof Campbell's(1957) validity typology and its extensions(Gadenne,1976;Kruglanski& Kroy, 1.976;Hultsch
6cHickey,1978;Cron'We
bach, 1982; Cronbachet al., 1980).
start first with two criticismsof internal
validity raisedby Cronbach(1982)and to a lbsserextentby Kruglanskiand Kroy
(1'976):(1) an atheoreticallydefinedinternal validity (A causesB) is trivial without referenceto constructs;and (2) causationin singleinstancesis impossible,includingin singleexperiments.
vALtDtrY nol
I
lnternal Validity ls Trivial
Cronbach(L982)writes:
I consider it pointless to speak of causeswhen all that can be validly meant by refermaenceto a causein a particular instanceis that, on one trial of a partially specified
phenamed,
not
conditions
other
nipulation under.orrditior6 A, B, and c, along with
nomenon p was observed.To introduce the word cause seemspointless. Campbell's
writings make internal validity a property of trivial, past-tense'and local statements'
(p .t3 7 )
Hence,.,causallanguageis superfluous"(p. 140).Cronbachdoesnot retaina specific role fo, .",rr"Iinferenceln his validity typology at all. Kruglanski and Kroy
(1976)criticizeinternalvalidity similanlSsaying:
are
The concrete events which constitute the treatment within a specific research
is
simply
it
Thus,
'
'
'
category'
meaningful only as members of a general conceptual
are
impossibleto draw strictly specificconclusionsfrom an experiment: our concepts
g.rr.r"l and each pr.r,rppor"s an implicit general theory about resemblanceberween
different concretecases.(p. 1'57)
Objections
to Causation
in SingleExperiments
A second criticism of internal validity deniesthe possibility of inferring causation
in a single experiment. Cronbach (1982) says that the important feature of causation is the "progressivelocalizationof a cause" (Mackie, 1974, p.73) over mul-
vALrDrry otu
|
tiple experimentsin a program of researchin which the uncertainties about the essential i."t.rr.r of the cause are reduced to the point at which one can characterize exacflywhat the causeis and is not. Indeed, much philosophy of causation asserts that we only recognize causes through observing multiple instances of a
putative causal relationship, although philosophers differ as to whether the mechanism for recognition involves logical laws or empirical regularities (Beauchamp,
1974;P. White, 1990).
However, some philosophers do defend the position that causescan be inMadden & Humferred in singleinstances(e.g.,Davidson, 1,967;Ducasse'1,95L1'
(e.g.,
Honore, 1985)'
Hart
&
law
in
the
ber, L97'1,).A good example is causation
by which we judge whether or not one person, say, caused the death of another
despitethe fact that the defendant may 4ever before have been on trial for a crime.
The verdict requires a plausible casethat (among other things) the defendantb actions precededlhe death of the victim, that those actions were related to the death,
that other potential causesof the death are implausible, and that the death would
not have occurred had the defendant not taken those actions-the very logic of
causal relationships and counterfactualsthat we outlined in Chapter 1. In fact, the
defendant'scriminal history will often be specifically excluded from consideration
in iudging guilt during the trial. The lessonis clear. Although we may learn more
"bo,rt ."nsation from multiple than from single experiments, we can rnf.ercause
in single experiments.Indeed, experimenterswill do so whether we tell them to or
not. Providing them with conceptual help in doing so is a virtue, not a vice; failing to do so is a major flaw in a theory of cause-probing methods.
Of course, individual experiments virtually always use prior concepts from
other experiments.However, such prior conceptualizations are entirely consistent
with the claim that internal validity is about causal claims in single experiments.
If it were not (at least partly) about single experiments, there would be no point
to doing the experiment, for the prior conceptualization would successfullypredict what will be observed.The possibility that the data will not support the prior
conceptualization makes internal validity essential.Further, prior conceptualizations are not logically necessary;we can experiment to discover effects that we
"The physicist George Darwin used
have no prior conceptual structure to expect:
to say tliat once in a while one should do a completely crazy experiment, like
blowing the trumper to the tulips every morning for a month. Probably nothing
wiil hafpen, but if something did happen, that would be a stupendousdiscovery"
(Hacking, L983, p. 15a). But we would still need internal validity to guide us in
judging if the trumpets had an effect.
466
ra.n cRrrcALAssEssMENT
oFouRAssuMproNs
reciprocalcausation.In contrast,mostwho write aboutexperimentationtoday espousetheoriesof probabilisticcausationin which the many difficultiesassociated
Even
with identifyingdependablecausalrelationshipsare humbly acknowledged.
languagethemselves,
more important, thesecriticsinevitablyusecausal-sounding
"mutual
"cause"
simultaneousshaping" (Lincoln 6c
for example,replacing
with
p.
seem
to us to avoidthe word but keep
Guba, 1985, 151).Thesereplacements
the concept,and for good reason.As we saidat the end of ChapterL, if we wiped
the slatecleanand constructedour knowledgeof the world aneq we believewe
would end up reinventingthe notion of descriptivecausationall over again, so
greatlydoesknowledgeof causeshelp us to survivein the world.
Between
ObjectionsConcerning
the Discrimination
ConstructValidityand ExternalValidity
Although we traced the history of the present validity system briefly in Chapter 2,
readers may want additional historical perspectiveon why we made the changes
we made in the present book regarding construct and external validity. Both
Campbell (1957) and Campbell and Stanley(1963) only usedthe phraseexternal
validitS which they defined as inferring to what populations, settings,treatment
variables, and measurement variables an effect can be generalized.They did not
rcfer at all to construct validity. However, from his subsequentwritings (Campbell, 1986), it is clear Campbell thought of construct validity as being part of external validity. In Campbell and Stanley therefore, external validity subsumed
generalizing from researchoperations about persons, settings,causes,and effects
for the purposes of labeling theseparticulars in more abstract terms, and also generalizing by identifying sourcesof variation in causal relationships that are attributable to person, setting, cause, and effect factors. All subsequentconceptualizations also share the same generic strategy based on sampling instancesof persons,
settings, causes,and effects and then evaluating them for their presumed correspondenceto targets of inference.
In Campbell and Stanley'sformulation, person, setting, cause,and effect categories share two basic similarities despite their surface differences-to wit, all of
them have both ostensive qualities and construct representations.Populations of
persons or settings are composed of units that are obviously individually ostensive. This capacity to point to individual persons and settings, especially when
they are known to belong in a referent category permits them to be readily enumerated and selectedfor study in the formal ways that sampling statisticiansprefer. By contrast, although individual measures (e.g., the Beck Depression Inventory) and treatments (e.g., a syringe full of a vaccine) are also ostensive,efforts to
enumerate all existing ways of measuring or manipulating such measuresand
treatments are much more rare (e.g.,Bloom, L956; Ciarlo et al., 1986; Steiner&
Gingrich, 2000). The reason is that researchersprefer to use substantivetheory to
determine which attributes a treatment or outcome measureshould contain in any
.J
vALrDtrYI oe,
given studS recognizing that scholars often disagreeabout the relevant attributes
of th. higher order entity and of the supposed best operations to representthem.
None of ihis negatesthe reality that populations of persons or settingsare also defined in part by the theoretical constructs used to refer to them, just like treatments
and outiomes; they also have multiple attributes that can be legitimately con'!(hat,
for instance, is the American population? \7hile a legal definition
tested.
surely exists,it is not inviolate. The German conception of nationality allows that
the gieat grandchildren of a German are Germans even if their parents and grandp"r*t, have not claimed German nationality. This is not possible for Americans.
And why privilege alegaldefinition? A cultural conception might admit as American all thor. illegal immigrants who have been in the United Statesfor decades
and it might e*cl.rde those American adults with passports who have never lived
in the United States. Given that person's,settings, treatments, and outcomes all
have both construct and ostensive qualities, it is no surprise that Campbell and
Stanley did not distinguish between construct and external validity.
Cook and Camptell, however, did distinguish between the two. Their unstated rationale for the distinction was mostly pragmatic-to facilitate memory
for the very long list of threats that, with the additions they made' would have
had to fit under bampbell and Stanley'sumbrella conception of external validity.
In their theoretical diicussion, Cook and Campbell associatedconstruct validity
with generalizingto causesand effects, and external validity with generalizing to
and across persons, settings, and times. Their choice of terms explicitly referencedCronbach and Meehl (1955) who used construct and construct validity in
"about higher-order constructs from remeasurementtheory to justify inferences
search operations'; lcook & Campbel| 1,979, p. 3S). Likewise, Cook and
Campbeli associatedthe terms population and external ualidity with sampling
theory and the formal and purposive ways in which researchersselect instances
of persons and settings. But to complicate matters, Cook and Campbell also
"all aspectsof the researchrequire naming samples in
brlefly acknowledged that
termi, including samplesof peoples and settings as well as samples
gener-alizable
of -r"r,rres or manipulations" (p. 59). And in listing their external validity
threats as statistical inieractions between a treatment and population, they linked
external validity more to generalizing across populations than to generalizing to
them. Also, their construct validity threats were listed in ways that emphasized
generalizing to cause and effect constructs. Generalizing across different causes
ind effect, *", listed as external validity becausethis task does not involve attributing meaning to a particular measure or manipulation. To read the threats
in Cook and Campbell, external validity is about generalizing acrosspopulations
of persons and settings and across different cause and effect constructs, while
construct validity is about generalizing to causesand effects.Where, then, is genera\zing from samples of persons or settings to their referent populations? The
text disiussesthis as a matter of external validitg but this classification is not apparent in the list of validity threats. A system is neededthat can improve on Cook
and Campbell's partial confounding between objects of generalization (causes
468
ASSESSMENT
OF OURASSUMPTIONS
14.A CRITICAL
and effects versus persons and settings) and functions of generalization (generalizing to higher-order constructs from researchoperations versus inferring the de-
,i-{11
f i..
t
.,.J
vALrDrrY
I oo,
tributes may be pertinent labels for some of the inferencesbeing made. Hence, we
usually have to rely on the assumption that, becausegender samplescome from
the same physical setting, they are comparable on all background characteristics
that might be correlated with the outcome. Becausethis assumption cannot be
fully testedand is ^nyw^y often false-as in the hypothetical example above-this
means rhat we could and should measure all the potential confounds within the
limits of our theoretical knowledge to suggestthem, and that we should also use
these measuresin the analysis to reduce confounding.
Even with acknowledged confounding, sample-specific differences in effect
sizesmay still allow us to conclude that a causal relationship varies by something
associatedwith gender.This is a useful conclusion for preventing premature overgeneralization.Iilith more breakdownq, confounded or not, one can even get a
senseof the percentageof contrastsacrosswhich a causal relationship does and
does not hold. But without further work, the populations across which the relationship varies are incompletely identified. The value of identifying them better is
particularly salient when some effect sizescannot be distinguished from zero. Although this clearly identifies a nonuniversal causal relationship, it does not advance theory or practice by specifying the labeled boundary conditions over which
a causal relationship fails to hold. Knowledge gains are also modest from generalization strategiesthat do not explicitly contrast effect sizes.Thus, when different populations are lumped together in a single hypothesis test, researcherscan
learn how large a causal relationship is despite the many unexamined sources of
variation built into the analysis. But they cannot accurately identify which constructs do and do not co-determine the relationship's size. Construct validity adds
useful specificity to external validity concerns, but it is not a necessarycondition
for external validity.'We can generalize across entities known to be confounded'
albeit lessusefully than acrossaccurately labeled entities.
This last point is similar to the one raised earlier to counter the assertion of
Gadenne (L9761and Kruglanski and Kroy (1976) that internal validity requires
the high consrruct validity of both causeand effect. They assertthat all scienceis
"something causedsomeabout constructs, and so it has no value to conclude that
thing sfss"-1hs result that would follow if we did a technically exemplary randomized experiment with correspondingly high internal validity but the causeand
effect were not labeled. Nonetheless, a causal relationship is demonstrably en"something reliably causedsomething else" might lead
tailed, and the finding that
to further researchto refine whatever clues are available about the cause and effect constructs. A similar argument holds for the relationship of construct to external validity. Labels with high construct validity are not necessaryfor internal
or for external validity, but they are useful for both.
Researchersnecessarilyuse the language of constructs (including human and
setting population ones) to frame their research questions and selecttheir representationsof constructsin the samplesand measureschosen.If they have designed
their work well and have had some luck, the constructs they begin and end with
will be the same,though critics can challengeany claims they make. However, the
470
OFOURASSUMPTIONS
ASSESSMENT
14.A CRITICAL
samplesand constructs might not match we[], and then the task is to examine the
samples and ascertain what they might alternatively stand for. As critics like
equivalent,so that theypresumablytap both into differentconstructsand into different causalrelationships.Either the samecausalconstructis differentlyrelated
to what now must be seenas two distinct outcomes,or the sameeffectconstruct
is differently related to two or more unique causal agents.So the intention to promote the construct validity of causesand effects by using multiple operations has
now facilitated conclusions about the external validiry of causesor effects;that is,
when the external validity of the causeand effect are in play, the data analysishas
revealed that more than one causal relationship needsto be invoked.
FortunatelS when we find that a causal relationship varies over different causes
or different effects, the research and its context often provide clues as to how the
causalelementsin eachrelationshipmight be (re)labeled.For example,the researcher
will generally examine closely how the operations differ in their particulars, and will
also study which unique meaningshave been attached to variants like thesein the ex-
isting literature.While the meaningsthat are achievedmight be lesssuccessfulbecause they have been devised post hoc to fit novel findings, they may in some crcumstances still attain an acceptable level of accuracy and will certainly prompt
continued discussion to account for the findings. Thus, we come full circle. I7e began with multiple operational representations of the same causeor effect when testing a single causal relationship; then the data forced us to invoke more than one relationship; and finally the pattern of the outcomes and their relationship to the
existing literature can help improve the labeling of the new relationships achieved.A
construct validity exercise begets an externat validity conclusion that prompts the
need for relabeling constructs. Demonstrating effect size variation acrossoperations
presumed to represent the same cause or effect can enhance external validity by
vALlDlrY I ort
ASSESSMENT
OFOURASSUMPTIONS
472 I T+.N CRITICAL
vALlDlw | 473
ulation who were not randomly sampledfall into the latter category.Nothing
requires
about externalvalidity,eitherin the presentbook or in its predecessors,
that all possibleuariuiion, of externalvalidity interestactuallybe observedin the
study-indeed, it would beimpossibleto do so,and we providedseveralarguments
in Cirapter2 aboutwhy it would not be wise to limit external validity questions
only to variationsactuallyobservedin a study.Of course,in most casesexternal
to things that were not studied are difficult, having to rely
ualidiry generalizations
on the .L.r..pt, and methodswe outlined in our grounded theory of generalized
causalinferencein Chapters11 through 13. But it is the great beautyof random
samplingthat it guaran;es that this generalizationwill hold over both sampledand
,rnr"-pl".d p.rr6nr. So it is indeedan externalvalidity questionwhah-e1acausal
relationshipthat hasbeenobservedin a singlerandomsamplewould hold for those
units that were in the populationbut not'in the random sample.
Inthe end,this book treatsthe labelingof a singlesampleof personsor settings asa matterof constructvalidiry whetheror not random samplingis used.It
alsi treatsthe generalizationof causalrelationshipsfrom a singlesampleto unobservedinstancesasa matterof externalvalidity-againrwhether or not random
samplingwas used.The fact that random sampling(which is associatedwith ex,.rrr"l uiiairy in this book) sometimeshappensto facilitatethe constructlabeling
of a sampleis incidentalto the fact that the population label is alreadyknown.
Though many populationlabelsare indeedwell-known, many more are still mat,.r, of debate,as reflectedin the exampleswe gavein Chapter3 of whetherpersonsshouldbe labeledschizophrenicor settingslabeledas hostilework environments.In theselatter cases,random samplingmakesno contribution to resolving
debatesabout the applicabilityof thoselabels.Instead,the principlesand methods we outlinedin Ci"pt.rs 11 through 13 will haveto be brought to bear.And
when random samplinghasnot beenused,thoseprinciplesand methodswill also
haveto be broughito b.". on the externalvalidity problemof generalizingcausal
relationshipsfrom singlesamplesto unobservedinstances.
of the Typology
ObjectionsAbout the Completeness
The first objectionof this kind is that our lists of particularthreatsto validity are
incomplete.Bracht and Glass(1,968),for example,ad-dednew externalvalidity
and
threatsthat they thought were overlookedby Campbelland Stanley(1,96311'
These
more recentlyAiken ind West (1991) pointed to new reactivity threats._
"r. i*portant becausethe key to the most confidentcausalconclusions
challenges
in our ,f,.ory of validity is the ability to construct a persuasiveargumentthat every
plausibleand identifiedthreat to validity has beenidentifiedand ruled out. Howiver, thereis no guaranteethat all relevantthreatsto validity havebeenidentified.
Our lists are not divinely ordained,as can be observedfrom the changesin the
threats from Campbel IUST) to Campbell and Stanley (1'963)to Cook and
14.A CRITICAL
ASSESSMENT
OF OURASSUMPTIONS
Campbell(1979) to this book. Threatsare better identifiedfrom insiderknowledgethan from abstractand nonlocal lists of threats.
A secondobjectionis that we may haveleft out particularvalidity fypesor organizedthem suboptimally.Perhapsthe bestillustration that this is true is Sackett's(1979) treatmentof bias in case-controlstudies.Case-controlstudiesdo not
designs;
commonly fall under the rubric of experimentalor quasi-experimental
but they are cause-probingdesigns,and in that sensea generalinterestin generalized causalinferenceis at leastpartly shared.Yet Sackettcreateda different typology.He organizedhis list around sevenstagesof researchat which biascan oc(3) in
and selection,
cur: (1) in readingaboutthe field, (2) in samplespecification
'in
measuringexposureand outcome,
defining the experimentalexposure,(4)
(5) in dataanalysis,(5) in interpretationof analyses,
and (71inpublishingresults.
Each of thesecould generatea validiry type, someof which would overlapconsiderablywith our validity types.For example,his conceptof biases"in executing
the experimentalmanoeuvre" (p. 62) is quite similar to our internal validiry
whereashis withdrawal biasmirrors our attrition. However,his list alsosuggests
new validity types,such as biasesin readingthe literature,and biaseshe lists at
each stageare partly orthogonal to our lists. For example,biasesin readinginclude biasesof rhetoric in which "any of severaltechniquesare usedto convince
the readerwithout appealingto reason"(p. 60).
In the end,then, our claim is only that the presenttypologyis reasonablywell
informed by knowledgeof the nature of generalizedcausalinferenceand of some
of the problemsthat are frequentlysalientabout thoseinferencesin field experimentation.It can and hopefullywill continueto be improvedboth by addition of
threatsto existing validity types and by thoughtful exploration of new validity
typesthat might pertainto the problem of generalizedcausalinferencethat is our
main concern.t
1. We are acutelyaware of, and modestlydismayedat, the many differentusagesof thesevalidity labelsthat have
developedover the years and of the risk that posesfor terminological confusion---eventhough we are responsible
for rnany of thesevariations ourselves.After all, the understandingsof validiry in this book differ from those in
Campbelland Stanley(1963),whoseonly distinctionwas betweeninternal and externalvalidity. They alsodiffer
from Cook and Campbell (7979), in which externalvalidity was concernedwith generalizingto and across
populations of personsand settings,whereasall issuesof generalizingfrom the causeand effect operations
constitutedthe domain of constructvalidity. Further,Campbell(1985) himselfrelabeledinternalvalidiry and
external validiry as local molar causalvalidity and the principle of proximal similarity, respectively.Steppingoutside
Campbell'stradition, Cronbach(1982) usedtheselabelswith yet other meanings.He said internalvalidity is the
problem of generalizingfrom samplesto the domain about which the questionis asked,which soundsmuch like our
construct validity except that he specifically denied any distinction betweenconstruct validiry and external validiry,
using the latter term to refer to generalizingresults to unstudied populations, an issueof extrapolation beyond the
data at hand. Our understandingof external validity includessuch extrapolations as one case,but it is not limited to
that becauseit also has to do with empirically identifying sourcesof variation in an effect sizewhen existing data
allow doing so. Finally, many other authors have casually used all theselabels in completelydifferent ways (Goetz
& LeCompte,1984; Kleinbaum,Kupper, & Morgenstern,1982;Menard, 1991).So in view of all thesevariations,
we urge that theselabels be used only with descriptionsthat make their intended understandingsclear.
::j
!t
VALIDTTY
47s
14.A CRITICAL
ASSESSMENT
OF OURASSUMPTIONS
beencited,for example,in U.S.SupremeCourt casesaboutallegedmisusesof testing practices (e.g., Albermarle Paper Co. v. MoodS 1975; Washington v. Davis,
i'
,;;::
"asonewaytojustifytheuseofatest"
word ualiditymoreexpansivelyforexample,
(Cronbach,1989,p. M9).It is only a short distancefrom validatinguseto validating action, becausemost of the relevantuseswere actionssuchas hiring or firing
someoneor labelingsomeoneretarded.Actions,in turn, haveconsequences-some
positive,suchas efficiencyin hiring and accuratediagnosisthat allows bettertailoring of treatment,and somenegative,suchas lossof incomeand stigmatization.So
Messick(1989, 1995l proposedthat validationalsoevaluatethoseconsequences,
especiallythe socialjusticeof consequences.
of test
Thus evaluatingthe consequences
usebecamea key featureof validity in test theory.The net resultwas a blurring of
the line betweenvalidity-as-truth and validity-as-evaluation,to the point where
Cronbach(1988)said"Validationof a testor testuseis evaluation"(p.4).
'We
strongly endorse the legitimacy of questions about the use of both tests and
experiments. Although scientistshave frequently avoided value questions in the mistaken belief that they cannot be studied scientifically or that scienceis value free, we
cannot avoid values even if we try. The conduct of experiments involves values at
every step, from question selection through the interpretation and reporting of results. Concerns about the usesto which experiments and their results are put and the
value of the consequencesof those usesare all important (e.g.,Shadishet al., 1991),
as we illustrated in Chapter 9 in discussingethical concerns with experiments.
However, if validity is to retain its primary association with the truth of
knowledge claims, then it is fundamentally impossible to validate an action becauseactions are not knowledge claims. Actions are more properly evaluated, not
validated. Supposean employer administers a test, intending to use it in hiring decisions. Suppose the action is that a person is hired. The action is not itself a
knowledge claim and therefore cannot be either true or false. Supposethat person
then physically assaultsa subordinate. That consequenceis also not a knowledge
claim and so also cannot be true or false. The action and the consequencesmerely
exist; they are ontological entities, not epistemological ones. Perhaps Messick
(1989) really meant to ask whether inferencesabout actions and consequencesare
true or false. If so, the inclusion of action in his (1,989)definition of validity is entirely superfluous, for validity-as-truth is already about evidencein support of inferences,including those about action or consequ.rr..s.'
2. Perhapspartly in recognitionof this, the most recentversionof the test standards(AmericanEducational
ResearchAssociation,American PsychologicalAssociation,and National Council on Measurementin Education,
1999) helpsresolvesomeof the problemsoudined hereinby removingreferenceto validatingaction from the
definition of validity: "Validity refersto the degreeto which evidenceand theory support the interpretationsof test
scoresentailedby proposedusesof tests" (p. 9).
i
,l
,I
t
VALIDITY I 477
478
ra. n cRrTrcAL
ASSESSMENT
OFOURASSUMPTTONS
notes that his intention is not to open the door to the social policing of truth (i.e.,
a test is valid if its social consequencesare good), but ambiguity on this issuehas
nonethelessopened this very door. For example, Kirkhart (1,995)cites Messick as
justification for judging the validity of evaluations by their social consequences:
"Consequential
validity refers here to the soundnessof changeexerted on systems
by evaluationand the extent to which thosechangesare just" (p.a).This notion is
risky becausethe most powerful arbiter of the soundnessand iustice of social consequencesis the sociopolitical systemin which we live. Depending on the forces in
power in that system at any given time, we may find that what counts as valid is
effectively determined by the political preferencesof those with power.
Validity in the Qualitative
Traditions
One of the most important developmentsin recent social researchis the expanded
use of qualitative methods such as ethnography ethnology, participant observation, unstructured interviewing, and case study methodology (e.g., Denzin 6c
Lincoln, 2000). These methods have unrivaled strengths for the elucidation of
meanings, the in-depth description of cases,the discovery of new hypotheses,and
the description of how treatment interventions are implemented or of possible
causal explanations. Even for those purposes for which other methods are usually
preferable,such as for making the kinds of descriptivecausalinferencesthat are the
topic of this book, qualitative methods can often contribute helpful knowledge and
'S7henever
reon rare occasionscan be sufficient (Campbell, 1975; Scriven, 1976ll.
sources allow, field experiments will benefit from including qualitative methods
both for the primary benefits they are capable of generatingand also for the assistance they provide to the descriptive causal task itself. For example, they can uncover important site-specificthreats to validiry and also contribute to explaining
experimental results in general and perplexing outcome patterns in particular.
However, the flowering of qualitative methods has often been accompanied
by theoretical and philosophical controversy, often referred to as the qualitativequantitative debates. These debates concern not just methods but roles and rewards within science,ethics and morality and epistemologiesand ontologies. As
part of the latter, the concept of validity has receivedconsiderableattention (e.g.,
Eisenhart & Howe, 1992; Goetz & LeCompte,1984; Kirk & Miller, 1'986;Kvale,
1.989;J. Maxwell, 1.992;J. Maxwell 6c Lincoln, 1.990;Mishler, 1,990;Phillips,
'Wolcott,
1,987;
1990). Notions of validity that are different from ours have occasionally resulted from qualitative work, and sometimesvalidity is rejectedentirely.
However, before we review those differences we prefer to emphasize the commonalities that we think dominate on all sides of the debates.
Comtnonalities. As we read it, the predominant view among qualitative theorists
is that validity is a concept that is and should be applicable to their work..We start
with examples of discussionsof validity by qualitative theorists that illustrate these
similarities becausethey are surprisingly more common than someportrayals in the
:
I
:l{
VALIDITYI O''
14.A CRITICAL
ASSESSMENT
OF OURASSUMPTIONS
I
I
:J
:iil
14.A CRITICAL
ASSESSMENT
OF OURASSUMPTIONS
.il
j
t
'iil
VALIDITYI +ET
-T
8z
oF ouR AssuMploNs
I r+.n cRtlcAL AssEssMENT
3. The fact that different people might have different beliefs about the same referent is sometimes cited as violating
this maxim, but it need not do so. For example, if the knowledge claim being validated is "John views the program
as effective but Mary views it as ineffective," the claim can be true even though the views of John and Mary are
contradictory.
ii
VALIDITY I 483
484
14.A CR|T|CAL
ASSESSMENT
OF OURASSUMPTTONS
Q UASI.EXPERIM ENTATION
Criteriafor RulingOut Threats:
The Centralityof FuzzyPlausibility
In a randomized experiment in which all groups are treated in the sameway excepr
for treatment assignment,very few assumptionsneed to be made about ro,rr.", of
bias. And those that are made are clear and can be easily tested,particularly as concerns the fidelity of the original assignment process and its subsequentmaintenance. Not surprisinglS statisticiansprefer methods in which the assumptionsare
few, transparent, and testable. Quasi-experiments, however, rely heavily on researcheriudgments about assumptions, especiallyon the fuzzy but indispensable
concept of plausibility. Judgments about plausibility are neededfor deciding which
of the many threats to validity are relevant in a given study for deciding whether
a particular designelement is capable of ruling out a given threat, for estimating by
how much the bias might have been reduced, and for assessingwhether multiple
threats that might have been only partially adjusted for might add up to a total bias
greater than the effect size the researcher is inclined to claim. Vith quasiexperiments, the relevant assumptions are numerous, their plausibility is less evident, and their single and joint effectsare lesseasily modeled. We acknowledgethe
fuzzy way in which particular internal validity threats are often ruled out, and it is
becauseof this that we too prefer randomized experiments (and regressiondiscontinuity designs)over most of their quasi-experimentalalternatives.
But quasi-experiments vary among themselveswith respect to the number,
transparencg and testability of assumptions. Indeed, we deliberately ordered the
chapters on quasi-experiments to reflect the increase in inferential power that
comes from moving from designs without a pretest or without a comparison
group to those with both, to those based on an interrupted time series,and from
there to regression discontinuity and random assignment.Within most of these
chapters we also illustrated how inferencescan be improved by adding design elements-more pretest observation points, better stable matching, replication and
systematic removal of the treatment, multiple control groups, and nonequivalent
dependentvariables. In a sense,the plan of the four chapters on quasi-experiments
reflects two purposes. One is to show how the number, transparency and testability of assumptions varies by type of quasi-experimental design so that, in the
best of quasi-experiments,internal validity is not much worse than with the randomized experiment. The other is to get students of quasi-experimentsto be more
sparing with the use of this overly general label, for it threatens to tar all quasi-
+SS
QUASI-EXPERIMENTATION
|
to the
experimentswith the samenegativebrush. As scholarswho have contributed
institution alization of the t i^ quoti-experiment, we feel a lot of ambivalence
the randomabout our role. Scholarsneed to itrint critically about alternatives to
laized experiment, and from this need arisesthe need for the quasi-experimental
under the
bel. But all instancesof quasi-experimentaldesignshould not be brought
do
studies
best
the
sameunduly broad quasi-experimentalumbrella if attributes of
not closely match the weaker attributes of the field writ large.
use of
Statisticians seek to make their assumptions transparent through the
stratthis
resisted
have
formal models laid out as formulae. For the most part, we
very conegy becauseit backfires with so many readers,alienating them from the
inwords
.!pt.r"t issuesthe formul ae aredesignedto make evident.'We have used
cognoscenti'
stead.There is a cost to this, and not jupt in the distaste of statistical
The
particularly those whose own research has emphasized statistical modelsformally
to
main cost is that our narrative approach makes it more difficult
the alternative
demonstrate how much fewer and more evident and more testable
quasiinterpretations became as we moved from the weaker to the stronger
acrossthe
.*p.ri-.rrts, both within the relevant quasi-experimental chapters and
'We
regret this, but do not apologize for the accessibility we tried to
set of them.
Fortucreate by minimirirrg the use of Greek symbols and Roman subscripts.
to develop
nately, this deficit is not absolute, as both we and others have worked
in particmeth;ds that can be used to measurethe size of particular threats' both
and
2000)
Shadish,
1998;
ular studies(e.g.,Gastwirth et al., L994;Shadishet al.,
Posavac,6c
in sets of studiis (e.g.,Kazdin 6c Bass, 1989; Miller, Turner, Tindale,
our
Further,
& Putnam,t982\.
& Rubin,1,978;Willson
Dugoni,1,991;Ror."nitt.t
statistical
narrative approach has a significant advantage over a more narrowly
threats
emphasisii allows us to addressa broad er array of qualitatively different
therethat
to validitS threats for which no statistical measure is yet available and
quantification.
fore mighi otherwise be overlooked with too strict an emphasison
at all
Better to h"u. imprecise attention to plausibility than to have no attention
measured'
paid to many imptrtant threats just becausethey cannot be well
PatternMatchingas a ProblematicCriterion
about the desirabilityof imbuing
This book is more explicitthan its predecessors
a causalhypothesiswith multiple tistable implicationsin the data, providedthat
we
they servett reducethe viability of alternativecausalexplanations.In a sense'
assessment
u-sual
the
for
havesoughtto substitutea pattern-matchingme{rod-ology'We
do this not because
differ.
reliably
of wheth-era few means,oft.n only fwo,
num.o-pl.*ity itself is a desideratumin science.To the contrary,simpliciry in the
The simplicity
be, of questionsaskedand methodsusedis highly prizedin science.
well.
of ,arrjomized experimentsfor descriptivecausalinferenceillustratesthis
However,the samesimple circumstancedoes not hold with quasi-experiments.
With them. we haveassirtedthat causalinferenceis improvedthe more specific,
RANDOMIZED
EXPERIMENTS
This sectionlistsobjectionsthat havebeenraisedto doingrandomizedexperiments,
and our analysisof the more and lesslegitimateissuesthat theseobiectionsraise.
Experiments
CannotBe Successfully
lmplemented
Even a little exposure to large-scalesocial experimentation shows that treatments
are often improperly or incompletely implemented and that differential attrition
often occurs. Organizational obstaclesto experiments are many. They include the
reality that different actors vary in the priority they attribute to random assignment, that some interventions seem disruptive at all levels of the organization,
and that those at the point of service delivery often find the treatment requirements a nuisance addition to their aheady overburdened daily routine. Then
there are sometimes treatment crossovers,as units in the control condition adopt
or adapt components from the treatment or as those in a treatment group are exposed to some but not all of these same components. These criticisms suggestthat
the correct comparison is not between the randomized experiment and better
quasi-experiments when each is implemented perfectly but rather between the
randomized experiment as it is often imperfectly implemented and better quasiexperiments. Indeed, implementation can sometimes be better in the quasiexperiment if the decision not to randomize is based on fears of treatment degradation. This argument cannot be addressedwell becauseit dependson specifying
the nature and degree of degradation and the kind of quasi-experimental alternative. But taken to its extreme it suggeststhat randomized experiments have no
special warrant in field settings becausethere is no evidencethat they are stronger
than other designs in practice (only in theory).
But the situation is probably not so bleak. Methods for preventing and coping with treatment degradation are improving rapidly (seeChapter 10, this vol-
EXPERIMENTS
RANDOMIZED
I AAS
ExperimentationNeedsStrongTheoryand Standardized
TreatmentlmPlementation
rs
Many critics claim that experimentationis more fruitful when an intervention
is
details
basedon strongsubstantivetheory when implementationof treatment
when imfaithful to that theor5 when the rlsearchsettingis well managed,and
these
plementationdoes,roi uury much betweenunits' In many field experiments'
organiza'
conditions are not met. For example,schools arclarge, complex, social
iio"r *ith multiple programs,disputatiouspolitics, and conflicting stakeholder
well as
goals.Many progr"*, a"reimplementedvariablyacrossschooldistricts,as
of standard
f.ror, ..hoth, .Lrrroo-r, arri ,t.rdents.Therecan be no presumPli9n
1'977)'
implementationor fidelity to programtheory (Berman& Mclaughlin,
wellBut thesecriticismsur., i' fa-ct,misplaced.Experimentsdo not require
implementaspecifiedprogram theories,good program management,standard
a contrition, or treatmentsthat are tJtally ?aithful to theory' Experimentsmake.
makesa
bution when they simplyprobewhetheran intervention-as-implemented
preceding
marginal improvem.tttt.yord other backgroundvariability. Still, the
suggests
fa.tJ* can ieducestatisticalpower and so cloud causalinference.This
should:
experiments
that in settingsin which *or. of these conditions hold,
(L) uselargesamplesto detecteffects;(2) take painsto reducethe influenceof exmatraneousvariation either by designor through measurementand statistical
worth studynipulation; and (3) studyimplementationquality both as a variable
implement
i"g * its own right in oid.r to ascertainwhich settingsand providers
treatthl interventionbetterand asa mediatorto seehow implementationcarries
ment effectsto outcome.
EXPERIMENTS
RANDOMIZED
| *tt
'aloJ
lueururo.rdeJoru qf,ntu
e sdeld drrprlerrlpuJetur ql1qlv\ ur surerSord
rpuar
;o
lurluerelm puorq dpursrrd
-Jns eql ol pue qf,ntu oor drlPllu^
Ieurelur aztseqduraapreqr qf,Jeesar;o sururSord
ur pa8raureaABr{stsaSSnsdrolsrq lpql sassau>lea^\
IertuaraJuragr ol uouuane Bur
-llEf, orp arrrtaqrey .(rq8rlrodsaql uI erurl slr a^eq lsntu ad& drlPler d-rela)
/lpll
-EA
pnJlsuof,
Jo
JaAo
drrprlerr
lEuJalxa
IeuJalur 1o drerurrd eurlnoJ due -ro; 8ur1er
lou eJEeM'T lardu{J uI Jealr oPELua.&\se 'esJnoJIO 'parseSSnsarreqsrrlr.rr tsed
req.a'\sPeaJxadlrear8 sluaut-radxaeldrllnur Ja o senssrdrrpllel leuJalxa pug lrnrls
-uol qloq sserppeol dlneder aql.slsdleue-Eleruur dlrrap
lsoru aeso^4,se 1ng ,sans
-sI asJl{t qroq Sulssarpp" ur r.lf,EarperFrll e^Er{ slueurradxa
.paluerg
lpnprlrpur
'larrr dlfsapou sanssr,{rrprlerr
Ipuralxo puB lJnrlsuoJ r{foq sserppE ot r{f,rBrsar
srueJSord;o
dlpeder
agr qrr^\ pessarduneJEaA\ ,1ser1uocdg
letuaurr.ladxeIo
'8ur1uru r{uo^\ tanau aJu
lpql stJoapqJfarrnbar sluaurrradxeter{l tsa88nsol luet
-.rodur ool sr s{Jo.&\rpqra 1no Surpurg .sanqod
alouord
IErJospaseq-sseua^rpa}Ja
ol lue1Y\ol{1v\sJJelsrlaql Pue srolelsr8al asoql ro; d1-rrlnrrued 'cnerualqord eJoru
uala dlqeqo-rdaru sJel\supJeell tnoqtr^ saurl aturl Buol-opelep qrns .re8uep
IEar
P sI uollEluaurtradxa arntreruardq8noqlly 'sploq uorlenlrs atues aql lsorule pue
'uerSor4
ruaurdolartaq IooqJS aqr uuSaq rauoJ sauef arurs sread 0t sl lI .sra/\,s
'sloogrs peleJelalf,eue8aq
-uB ou
urle-J drua11
PUPsluolutradxa ou aleq all\ Pue
'sfteJJe
eruts sread SI sl fI
JIeI{l rnoqe sJa^.r{sup
Jeolf,ou e^Erl llrls o.&\puu .pasod
-ord arain sJarlf,no^
.splp .uoryo oor
Iooqtrs erurs srcad 0t A ou sr lI
IIE ,.raddeq
'elgBpuedapun
SFII
sI lEql uollf,auuoJ lusnpr E tnogp suorsnlf,uoo
leraua8 puorg
Sutmerp >lsIJol sI uolluelJetul uE Jo srlaJJear{l uo sarpnts
leluauuadxa Suorls o4
e^Pq ol 'spuno;8 IEIIuePI^ero lerrSoyuo elqrsneldurrdl-realf,arg drrprlerr
leuralur
ol slBarql ssaFn 'saf,uareJur
da4;o dtr.rSalureql Sursrulordtuot lnoqlrd\ passoJl eq
louupf, spunoq aruos 'lurod srql ot rrlaqledtuds d11erauafi
are am qfinoqrly .ftget
'qrequo.r3 :og5t ''1"
''3'a)
Ir r{luquorJ
spoqlau lutuaurradxo ra8uorls aqr Bur
-zrsuqdruasrue.rSo-rd
tuory ueql serpnls leluaunradxeuou pue Iuluaur-radxa_isenb
;o dlerrlua uela ro dlrsoru lslsuof, rer{l qf,reasarJo sure-r8ordtuory peuJuel aq IIri\,\
uoupluroJul
aroru
Et'*:u"';rrx;;H:;r:::ilil?
r'rrrpourr
InJasn
ilHt", ",
sNoll_dwnssv
uno lo l_Nty\sslssv
tv)tl|u) v .tt I zov
I
EXPERIMENTS
RANDOMIZED
| 493
I
Assumean InvalidModel
Experiments
Utilization
of Research
model of decision
To somecritics, experimentsrecreatea naive rational choice
among (the treatmaking. That is, one first lays out the alternativesto choose
one collectsin*.rr,rt] then one decideson criteria of merit (the outcomes);then
and finally
formation on eachcriterion for eachtreatment(the data collection),
empirical
one makes a decisionabout the superior alternative.UnfortunatelS
so simpleas the rawork on the useof socialsciencedaia showsthat useis not
(c. \ufeiss6c Bucuvalas,1980; c''weiss, 1988)'
tional choicemodelsuggests
contexts'exFirst, evenwhen."-rir. and effectquestionsare askedin decision
exp.ri-.nt"l resultsare still usedalong with other forms of information-from
consensusof a
isting theories,personaltestimony,extrapolationsfrom surveys'
haverecentlybefieldlchims from expertswith intereststo defend,and ideasthat
politics' person.o*. trendy.Decisionsare shapedpartly by ideology,interests,
as much made by a
ality, windows of-opportunity, and ualues;and they are
individualor compolicy-shapirrg.o-*nrrity (cronbachet al., 1980) as by an
overtime asear*i,,... Fuither,manydecisionsarenot so much madeasaccreted
maker with few oplier decision,.orrrir"in later ones,leavingthe final decision
are available,new
tions ('Weiss,1980). Indeed,by the time ixperimental results
decisionmakersand issuesmay havereplacedold ones.
verdicts
Second,.*p.rirn.nts often yield contestedrather than unanimous
Disputes arise about
that therefore have uncertain implications for decisions.
resultsare valid'
whether the causalquestionswere correctly framed, whether
and whetherthe resultsentail a specific
whetherrelevantoutcomeswere assessed,
voucher
decision.For example,reexaminationsof the Milwaukee educational
(H'
occurred
;"rdy offereddifferentconclusionsabout whetherand whereeffects
SimilarlS
1'998,"1'999,2000)'
6cDu, 1.999;'Sritte,
Fuller,2000;Greene,Peterson,
classsizeexperiment(Finn
differenteffect,ir., *.r. generatedfrom the Tennessee
Light, 6c Sachs,1996)'Sometimes,
EcAchilles,1.990;Hanusi'ek,1999;Mosteller,
are at issue,but at other timesthe disputesreflectdeeply
scholarlydisagreements
conflictedstakeholderinterests.
likely when
Third, short-terminstrumentaluseof experimentaldata is more
it is easierto
the interventionis a minor variant on existingpractice.For example,
criteriafor
changetextbooksin a classroomor pills givenlo patientsor eligibility
or to open
entry than it is to relocatehospitalsto.underservedlocations
;;;;"*
state' Becausethe
day-carecentersfor welfare recipientsthroughout an entire
to dramatically
more feasible.tt""g.t are so ,ood.r, in scope,they are lesslikely
on shor-t-terminaffecttheproble- ih.y address.So critics note that prioritizing
is unlikelyto solve
strumentalchangetendsto preservemost of the statusquo and
that truly twist
tr.rr.hunt social"probl.-s. bf course'thereare someexperiments
from denselypoor
the lion,stail andinvolvebold initiatives.Thus moving families
deviations
inner-citylocationsto the suburbsinvolveda changeof three standard
494
14.A CRIT|CAL
ASSESSMENT
OFOURASSUMPTTONS
in the poverty level of the sending and receiving communities, much greater than
what happens when poor families spontaneously move.'S7hethersuch a dramatic
change could ever be used as a model for cleaning out the inner cities of those who
want to move is a moot issue. Many would judge such a policy to be unlikely.
Truly bold experiments have many important rationales; but creating new policies
that look like the treatment soon after the experiment is not one of them.
Fourth, the most frequent use of research may be conceptual rather than instrumental, changing how users think about basic assumptions,how they understand contexts, and how they organize'or label ideas. Some conceptual uses are
intentional, as when a person deliberately reads a book on a current problem; for
example, Murray's (1984) book on social policy had such a conceptual impact in
the 1980s, creating a new social policy agenda. But other conceptual usesoccur
in passing, as when a person reads a newspaper story referring to social research.
Such usescan have great long-run impact as new ways of thinking move through
the system, but they rarely change particular short-term decisions.
These arguments against a naive rational decision-making model of experimental usefulnessare compelling. That model is rightly rejected. However, mosr
of the objections are true not just of experiments but of all social sciencemethods.
Consider controversies over the accuracy of the U.S. Census,the entirely descriptive results of which enter into a decision-making process about the apportionment of resourcesthat is complex and highly politically charged. No method offers a direct road to short-term instrumental use. Moreover, the obiections are
exaggerated.In settings such as the U.S. Congress,decision making is sometimes
influenced instrumentally by social scienceinformation (Chelimsky, 1998), and
experiments frequently contribute to that use as part of a researchreview on effectivenessquestions. Similarlg policy initiatives get recycled, as happened with
school vouchers, so that social science data that were not used in past years are
used later when they become instrumentally relevant to a current issue (Polsby,
1'984; Quirk, 1986).In addition, data about effectivenessinfluence many stakeholders' thinking even when they do not use the information quickly or instrumentally. Indeed, researchsuggeststhat high-quality experiments can confer exrra
'Weiss
credibility among policymakers and decision makers (C.
& Bucuvalas,
1980)' as happened with the Tennesseeclasssize study. We should also not forget
that the conceptual use of experiments occurs when the texts used to train professionalsin a given field contain results of past studies about successfulpractice
(Leviton 6c Cook, 1983). And using social sciencedata to produce incremental
change is not always trivial. Small changescan yield benefits of hundreds of millions of dollars (Fienberg,Singer,& Tanur, 1985). SociologistCarol'Weiss,an advocate of doing research for enlightenment's sake, says that 3 decadesof experience and her studies of the use of social sciencedata leave her "impressed with the
utility of evaluation findings in stimulating incremental increasesin knowledge
and in program effectiveness.Over time, cumulative incrementsare not such small
potatoes after all" ('Weiss,1998, p. 31,9).Finallg the usefulnessof experimentscan
be increased by the actions outlined earlier in this chapter that involve comple-
ott
EXPERIMENTS
RANDOMIZED
I
mentingbasicexperimentaldesignwith adjunctssuchas measuresof implemenprotation a-ndmediationo, qualitativemethods-anything that will help clarify
gram processand implementationproblems.In summarSinvalid modelsof the
commonthan
ir.foln.rs of experimintalresultsseemto us to be no more nor less
'we
have learned
invalid modelslf th. use of any other social sciencemethods.
their
much in the last severaldecadesabout use, and experimenterswho want
1'99I).
of thoselessons(Shadishet al.,
work to be usefulcan take advantages
Differfrom the
TheConditionsof Experimentation
Conditionsof Policylmplementation
if
were
Experimentsare often doneon a smalleiscalethan would pertain services
relei-il.-r.rted state-or nationwide,and so they cannot mimic all the details
intervenu"rr, ,o full policy implementation.Hencepolicy implementationof an
For ex,i"" -ry yi.ta aiff.rint o,rt.omesthan the experiment(Elmore, 1996)'
size,
class
ample, t"r.d partly on researchabout the benefits of reducing
and Caliiornia implementedstatewidepoliciesto have more classes
Tennessee
classwith fewer studentsin each.This required many new teachersand new
teachnew
those
of
rooms.However,becauseof a nationalteachershortage,some
of
ers may havebeenlessqualifiedthan thosein the experiment;and a shortage
have
may
that
classroomsled to more .rs. of trailers and dilapidatedbuildings
further.
harmedeffectiveness
enthuSometimesan experimentaltreatmentis an innovation that generates
experisiasticeffortsto implementit well. This is particularly frequentwhen the
that
ment is done by a charismaticinnovator whosetacit knowledgemay exceed
pr^ctrce
of thosewho would be expectedto implementthe program in ordinary
may
factors
These
and whosecharismamay inducehigh-qualityimplementation.
is imgeneratemore srr...srfoi outcomesthan will be seenwhen the intervention
plementedasroutine PolicY.
Policy implementationmay also yield different-resultswhen experimental
practreatmentsare implementedin a fashionthat differs from or conflictswith
psychotherapy
ticesin real-*orld application.For example,experimentsstudying
and
outcomeoften standardizetreatmentwiih a manual and sometimesobserve
but
correct the therapistfor deviatingfrom the manual (shadishet al., 2000);
effecthesepracticesare rare in clinicallractice. If manualizedtreatmentis more
might
tive (bhambless& Hollon, 1998; Kendall, 1998), experimentalresults
transferpoorly to practicesettings.
policy
Raniom assigrrm.ntmay also changethe program from the intended
implementation(i{eckman,l992l. For ixample, thosewilling to be randomized
may
-"y diff.r from those for whom the treatment is intended; randomizatLon
with
changepeople'spsychologicalor social responseto treatment compared
those"wlroself-selecttreatment;and randomizationmay disrupt administration
clients'
and implemenrationby forcingthe programto copewith a differentmix of
496
14.A CR|T|CAL
ASSESSMENT
OF OURASSUMPTIONS
Heckman claims this kind of problem with the Job taining PartnershipAct
"calls into question
0TPA) evaluation
the validity of the experimentalestimates
as a statementabout theJTPAsystemas a whole" (Heckman,1.992,
p. ZZ1,).
In many respects,we agreewith thesecriticisms,thoughit is worth noting several responsesto them. First, theyassumealack of generalizabllityfrom experiment to policy but that is an empirical question.Somedata suggesrthar generalization may be high despite differencesbetweenlab and field (C. Anderson,
LindsaS & Bushman, 1999) or betweenresearchand practice (Shadishet al.,
2000). Second,it can help to implement.treatment
underconditionsthat aremore
characteristicof practiceif it doesnot unduly compromiseother researchpriorities. A little forethoughtcan improve the surfacesimilarity of units, trearments,
observations,settings,or timesto their intendedtargets.Third, someof thesecriticismsare true of any researchmethodologyconductedin a limited context,such
as locally conductedcasestudiesor quasi-experiments,
becauselocal implementation issuesalwaysdiffer from large-scaleissues.Fourth, the potentiallydisruptive natureof experimentallymanipulatedinterventionsis sharedby many locally
'rrr"or"h
invented novel programs, euen uhen they are not studied by any
methodologyat all.Innovation inherentlydisrupts,and substantiveliteraturesare
rife with examplesof innovationsthat encounteredpolicy implementationimpediments(Shadish,1984).
However,the essentialproblem remainsthat large-scalepolicy implementation is a singularevent,the effectsof which cannot be fully known exceptby doing the full implementation.A singleexperiment,or evena smallseriesof ri-ilrt
ones,cannotprovidecompleteanswersabout what will happenif the intervention
is adoptedas policy. However,Heckman'scriticism needsreframing.He fails to
distinguishamongvalidity types(statisticalconclusion,internal,.onrtro.., external). Doing so makesit clearthat his claim that suchcriticism"calls into question
the validity of the experimentalestimatesasa sratementabout the JTPA,yrt.rr, ",
a whole" (Heckman,1.992,
p.221,)is reallyabout externalvalidityand construcr
validity,not statisticalconclusionor internalvalidity.Exceptin thenarrow econometricstraditionthat he understandably
cites(Haavelmo,7944;Marschak
,7953;
Tinbergen,1956),few socialexperimentersever claimedthat experimentscould
describethe "systemas a whole"-even Fisher(1935)acknowledged
this tradeoff. Further,the econometricsolutionsthat Heckman suggestscannot avoid the
sametradeoffsbetweeninternal and externalvalidity. For example,surveysand
certain quasi-experiments
can avoid someproblemsby observingexistinginterventionsthat have aheadybeenwidely implemented,but the validity of tleir estimatesof program effectsare suspectand may themselves
changeif the program
were imposedevenmore widely as policy.
Addressingthesecriticismsrequiresmultiple lines of evidence-randomized
experimentsof efficacyand effectiveness,
nonrandomizedexperimentsthat observeexistinginterventions,nonexperimentalsurveysto yield estimatesof representativeness,
statisticalanalysesthat bracketeffectsunder diverseassumpd;ns,
EXPERIMENTS
RANDOMIZED
II Ot
qualitative observation to discover potential incompatibilities between the interventiol and its context of likely implementation, historical study of the fates of
similar interventions when they were implemented as policg policy analysesby
those with expertisein the type of intervention at issue,and the methods for causal
generalizationin this book. The conditions of policy implementation will be difi.r.rr, from the conditions characteristic of any rese^rchstudy of it, so predicting
generalizationto policy will always be one of the toughest problems.
Flawed
ls Fundamentally
lmposingTreatments
the Growthof Local
with Encouraging
Compared
Solutionsto Problems
Experimentsimposetreatmentson recipients.Yet som,elate 20th-centurythought
,.rjg.rt, that imposedsolutionsmay be inferior to solutionsthat are locally gen.rJr".a by thoseiho h"n. the problem. Partly,this view is premisedon research
findings of few effectsfor the Great Societysocialprogramsof the 1960sin the
UniteJ States(Murrag 1.984;Rossi, L987),with the presumptionthat a portion
of the failurewas due to the federallyimposednatureof the programs.Partly,the
view reflectsthe successof late 2Oth-centuryfree market economicsand conservative political ideologiescompared with centrally controlled economiesand
more fi|eral political beliefs.Experimentallyimposedtreatmentsare seenin some
quartersas beinginconsistentwith suchthinking'
IronicallS the first objectionis basedon resultsof experiments-if it is true
that impos.i progr"*s do not work, experimentsprovided the evidence.Moreover,thesetro-.ff..t findingsmay havebeenpartly due to methodologicalfailures
of experimentsas they were implementedat that time. Much progressin solving
practicalexperimentalproblemsoccurredafter,and partly in responseto, those
If so,it is prematureto assumetheseexperimentsdefinitivelydemonexperiments.
stiated no effect,especlalygiven our increasedability to detectsmall effectsto6c shroder,1,997;LipseSL992;Lipsey6c'Wilson,!993).
day
' (D. Greenberg
iistinguish betweenpolitical-economiccurrencyand the effects
We must also'We
of interventions. know of no comparisonsof, say,the effectsof locally generatedversusimposedsolutions.Indeed,the methodologicalproblemsin doing such
comparisonsare daunting, especiallyaccuratelycategotizinginterventionsinto
the two categoriesand unlonfounding the categorieswith correlatedmethoddifferences.Bariing an unexpectedsolutionto the seeminglyintractableproblemsof
causalinferencein nonrandomizeddesigns,answeringquestionsabout the effects
of locally generatedsolutionsmay requireexactlythe kind of high-qualityexperimentatioi being criticized.Though it is likely that locally generatedsolutions
may indeedhavesignificantadvantages,it also is likely that someof thosesolutions will haveto be experimentallyevaluated.
CAUSALGENERALIZATION:
AN OVERLY
COMPLICATED
THEORY?
Internal validity is best promoted via random assignment,an omnibus mechanism
that ensuresthat we do not have many assumptions to worry about when causal inferenceis our goal. By contrast, quasi-experimentsrequire us to make explicit many
assumptions-the threats to internal validity-that we then have to rule out by fiat,
by design,or by measurement.The latter is a more complex and assumption-riddled
processthat is clearly inferior to random assignment.Something similar holds for
causal generalization,in which random selectionis the most parsimonious and theoretically justified method, requiring the fewest assumptionswhen causalgeneralization is our goal. But becauserandom selectionis so rarely feasible,one instead
has to construct an acceptabletheory of generaliz tion out of purposive sampling,
'We
a much more difficult process.
have tried to do this with our five principles of
generalizedcausal inference.These, we contend, are the keys to generalizedinference that lie behind random sampling and that have to be identified, explicated,
and
ano assessed
assessedif
rt we are to make
make better general inferences,
rnterences,even rt
if they are not perfect ones. But these principles are much more complex to implement than is random sampling.
Let us briefly illustrate this with the category called American adult women.
We could represent this category by random selection from a critically appraised
register of all women who live in the United Statesand who arc at least 21 years
of age.I7ithin the limits of sampling error, we could formally generalizeany characteristics we measured on this sample to the population on that register. Of
course, we cannot selectthis way becauseno such register exists.Instead,one does
onet experiment with an opportunistic sample of women. On inspection they all
'1,9
turn out to be between
and 30 years of age, to be higher than average in
achievementand abilit5 and to be attending school-that is, we have useda group
of college women. Surface similarity suggeststhat each is an instance of the category woman. But it is obvious that the modal American woman is clearly not a
college student. Such students constitute an overly homogeneoussample with respect to educational abilities and achievement,socioeconomicstatus, occupation,
and all observable and unobservable correlates thereof, including health status,
current employment, and educational and occupational aspirations and expectations. To remedy this bias, we could use a more complex purposive sampling design that selectswomen heterogeneouslyon all these characteristics.But purposive sampling for heterogeneousinstances can never do this as well as random
selection can, and it is certainly more complex to conceive and execute.I7e could
go on and illustrate how the other principles faclhtate generalization. The point is
that any theory of generalization from purposive samples is bound to be more
complicated than the simplicity of random selection.
But becauserandom selection is rarely possible when testing causal relationships within an experimental framework, we need these purposive alternatives.
NONEXPERIMENTALALTERNATIVES
I 499
Yet most experimental work probably still relies on the weakest of these alternatives, surfaci similarity.'We seek to improve on such uncritical practice. Unfortunately though, there is often restricted freedom for the more careful selection of
instancesof units, treatments, outcomes, and settings, even when the selection is
done purposively.It requires resourcesto sample irrelevanciesso that they are heterogeneouson many attributes, to measure several related constructs that can be
discriminated from each other conceptually and to measure a variety of possible
explanatory processes.This is partly why we expect more progress on causal generalization from a review context rather than from single studies. Thus, if one researcher can work with college women, another can work with female schoolteachers, and another with female retirees, this creates an opportunity to see if
thesesourcesof irrelevant homogeneity make a difference to a causal relationship
or whether it holds over all these differ6nt types of women.
UltimatelS causal generalizationwill always be more complicated than assessing the likelihood that a relationship is causal.The theory is more diffuse, more recent, and lesswell testedin the crucible of researchexperience.And in some quarters there is disdain for the issue,given the belief and practice that relationshipsthat
replicate once should be consideredas generaluntil proven otherwise' not to speak
oithe belief that little progressand prestigecan be achieved by designingthe next
experiment to be some minor variant on past studies. There is no point in pret.nding that causal generalization is as institutionalized procedurally as other
methods in the social sciences.'Wehave tried to set the theoretical agendain a systematic way. But we do not expect to have the last word. There is still no explication of causal generalizationequivalent to the empirically produced list of threats
to internal validiry and the quasi-experimental designsthat have evolved over 40
years to rule out thesethreats. The agendais set but not complete.
RIM ENTALALTERNATIVES
NONEXPE
Though this book is about experimentalmethodsfor answeringquestionsabout
it is a mistaketo believethat only experimentalapproachesare
.".rr"l hypotheses,
used for thir p,r.pose.In the following; we briefly consider severalother approaches,indiiating the major reasonswhy we havenot dwelt on them in detail.
basicallSthe reasonis that we believethat, whatevertheir merits for someresearchpurposes,they generatelessclearcausalconclusionsthan randomizedexor
suchas regression-discontinuity
perimentsor eventhe bestquasi-experiments
interruptedtime series
The nonexperimentalalternativeswe examineare the major onesto emerge
in variousacademicdisciplines.In educationand parts of anthropologyand sociologg one alternativeis intensivequalitativecasestudies.In thesesamefields,and
also-in developmentalpsychologythere is an emerginginterestin theory-based
500
14.A CR|T|CAL
ASSESSMENT
OFOURASSUMPTTONS
IntensiveQualitativeCaseStudies
The call to generate causal conclusions from intensive case studies comes from
several sources. One is from quantitative researchersin education who became
disenchanted with the tools of their trade and subsequently came to prefer the
qualitative methods of the historian and journalist and especiallyof the ethnographer (e.g.,Guba,198l, 1,990;and more tentatively Cronbach, 1986).Another
is from those researchersoriginally trained in primary disciplines such as qualitative anthropology (e.g.,Fetterman, 19841or sociology (Patton, 1980).
The enthusiasm for case study methods arises for several different reasons.
One is that qualitative methods often reduce enough uncertainty about causation
to meet stakeholderneeds.Most advocatespoint out that journalists,historians,
ethnographers, and lay persons regularly make valid causal inferences using a
qualitative processthat combines reasoning, observation, and falsificationist procedures in order to rule out threats to internal validity-even if that kind of language is not explicitly used (e.g.,Becker,1958; Cronbach,1982). A small minority of qualitative theorists go even further to claim that casestudiescan routinely
replace experiments for nearly any causal-sounding question they can conceive
(e.g.,Lincoln & Guba, 1985). A secondreasonis the belief that suchmethodscan
also engagea broad view of causation that permits getting at the many forces in
the world and human minds that together influence behavior in much more complex ways than any experiment will uncover.And the third reasonis the belief that
case studies are broader than experiments in the types of information they yield.
For example, they can inform readers about such useful and diverse matters as
how pertinent problems were formulated by stakeholders, what the substantive
theories of the intervention are, how well implemented the intervention components were, what distal, as well as proximal, effects have come about in respondents' lives, what unanticipated side effects there have been, and what processes
explain the pattern of obtained results.The claim is that intensivecasestudy methods allow probes of an A to B connection, of a broad range of factors conditioning this relationship, and of a range of intervention-relevant questions that is
broader than the experiment allows.
.J
NONEXPERIMENTALALTERNATIVES
| 501
I
Evaluations
Theory-Based
This approach has beenformulated relatively recently and is describedin various
books or specialjournal issues(Chen & Rossi, 1,992;Connell, Kubisch, Schorr,&
'Weiss,
1.995;Rogers,Hacsi, Petrosino,& Huebner, 2000). Its origins are in path
analysis and causal modeling traditions that are much older. Although advocates
have some differenceswith each other, basically they all contend that it is useful:
(1) to explicate the theory of a treatment by detailing the expected relationships
among inputs, mediating pfocesses,and short- and long-term outcomes; (2) to
measure all the constructs specified in the theory; and (3) to analyzethe data to
assessthe extent to which the postulated relationships actually occurred. For
shorter time periods, the available data may addressonly the first part of a postulated causal chain; but over longer periods the complete model could be involved. Thus, the priority is on highly specific substantive theorS high-quality
measurement,and valid analysisof multivariate explanatory processesas they unfold in time (Chen & Rossi, 1'987,1,992).
Such theoretical exploration is important. It can clarify general issueswith treatments of a particular type, suggestspecific researchquestions,describehow the intervention functions, spell out mediating processes,locate opportunities to remedy implementation failures, and provide lively anecdotesfor reporting results ('Weiss,1'998).
All th.r. serveto increasethe knowledge yield, evenwhen such theoretical analysisis
done within an experimental framework. There is nothing about the approach that
makes it an alternative to experiments. It can clearly be a very important adjunct to
such studies,and in this role we heartily endorsethe approach (Cook,2000).
However, some authors (e.g., Chen 6c Rossi, 1,987, 1992; Connell et al.,
1,995l have advocated theory-based evaluation as an attractive alternative to experiments when it comes to testing causal hypotheses.It is attractive for several
i.urorrr. First, it requires only a treatment group' not a comparison group whose
agreement to be in the study might be problematic and whose participation increasesresearchcosts. Second, demonstrating a match between theory and data
suggeststhe validity of the causal theory without having to go through a laborious processof explicitly considering alternative explanations. Third, it is often impractical to measure distant end points in a presumed causal chain. So confirmation of attaining proximal end points through theory-specified processescan be
used in the interim to inform program staff about effectivenessto date, to argue
for more program resourcesif the program seemsto be on theoretical track, to
justify claims that the program might be effective in the future on the as-yet-notassesseddistant criteria, and to defend against premature summative evaluations
that claim that an intervention is ineffective before it has been demonstrated that
the processesnecessaryfor the effect have actually occurred.
However, maior problems exist with this approach for high-quality descriptive causalinference(Cook, 2000). First, our experiencein writing about the theory of a program with its developer (Anson et al., 1,991)has shown that the theory is not always clear and could be clarified in diverse ways. Second, many
theories are linear in their flow, omitting reciprocal feedback or external contingenciesthat might moderate the entire flow. Third, few theories specify how long
it takes for a given processto affect an indicator, making it unclear if null results
disconfirm a link or suggestthat the next step did not yet occur. Fourth, failure to
corroborate a model could stem from partially invalid measuresas opposedto invalidity of the theory. Fifth, many different models can fit a data set (Glymour et
a1.,1987;Stelzl, 1986), so our confidencein any given model may be small. Such
problems are often fatal to an approach that relies on theory to make strong causal
claims. Though some of theseproblems are present in experiments (e.g.,failure to
incorporate reciprocal causation, poor measures),they are of far less import because experiments do not require a well-specified theory in constructing causal
knowledge. Experimental causal knowledge is less ambitious than theory-based
knowledge, but the more limited ambition is attainable.
Weaker Quasi-Experi
ments
For some researchers,random assignment is undesirable for practical or ethical
reasons, so they prefer quasi-experiments. Clearly, we support thoughtful use of
quasi-experimentation to study descriptive causal questions. Both interrupted
time series and regression discontinuity often yield excellent effect estimates.
Slightly weaker quasi-experiments can also yield defensible estimates,especially
when they involve control groups with careful matching on stable pretest attributes combined with other design features that have been thoughtfully chosen to
addresscontextually plausible threats to validity. However, when a researchercan
choose, randomized designsare usually superior to nonrandomized designs.
This is especially true of nonrandomized designs in which little thought is
given to such matters as the quality of the match when creating control groups,
NONEXPERIMENTALALTERNATIVES
I tOl
includingmultiple hypothesistestsrather than a singleone' generatingdata from
severalpr.tr."t*.nt time points rather than one, or having severalcomparison
groupsto createcontrolsthat bracketperformancein the treatmentgroups.Inare comparedwith thosefrom
I..d, when resultsfrom typical quasi-experiments
randomizedexperimentson the same topic, several findings emerge.Quasiexperimentsfrequentlymisestimateeffects(Heinsman& Shadish,1'996;Shadish
& Ragsdale,t9961.Tiresebiasesare often large and plausiblydue to selectionbiof more distressedclientsinto psychotherapytreatasessrrchas the self-selection
ment conditions(Shadishet al., 2000) or of patientswith a poorer prognosisinto
controlsin medicalexperiments(Kunz & Oxman,1'9981.Thesebiasesare espethat usepoor quality control groupsand have
cially prevalentin quasi-experiments
6cRagsdale,l996l.So,if the
higheiattrition(Heinsmar$cShadish,'1,996;Shadish
more
crediblethan thosefrom
obtainedfrom randomizedexperimentsare
an"swers
on theoreticalgroundsand are more accurateempirically,then
quasi-experiments
,'h. ".g,.r-entsfor randomizedexperimentsare evenstrongerwhenevera high degr.. oI uncertaintyreductionis requiredabout a descriptivecausalclaim.
are not equal in their ability to reduceuncerBecauseall quasi-experiments
tainty about."ur., *. -"ttt to draw attention againto a common but unfortuis beingdone
natepracticein manysocialsciences-tosaythat a quasi-experiment
in order to provide justificationthat the resultinginferencewill be valid. Then a
designis describedthat is so deficientin the desirablestrucquasi-experimental
tural featuresnoted previously,which promote better inference,that it is probanoted the term
bly not worth doing. Indeed,over the yearswe have__repeatedly
biing usedto justify designsthat fell into the classthat Campquasi-experiment
and that Cook and Campbell
bell and'stanley(196i) labeledas uninterpretable
Theseare the simplestforms of the
generallyuninterpretable.
(1,9791labeled'as
cannot be an alternadesignsdiscussedin Chapters4 and 5. Quasi-experiments
tive to randomizedexperimentswhen the latter are feasible,and poor quasi-exwhen_thelatperimentscan neverbi a substitutefor strongerquasi-experiments
i., "r. also feasible.Just as Gueron (L999) has remindedus about randomized
haveto be fought for, too. They are rarely
experiments,good quasi-experiments
handedout as though on a silverplate.
StatisticalControls
In this book,we haveadvocatedthat statisticaladjustmentsfor groupnonequivalence
are best urrd oBt designcontrolshavealreadybeenusedto the maximum in order
to a minimum. So we are not opponentsof statisticaladto reducenonequivalence
justmenttechniquessuchasthoseadvocatedby the statisticiansand econometricians
describedin the appendixto Chapter5. Ratheqwe want to usethem as the last resort.The positionwe do not like is the assumptionthat statisticalcontrolsare sowell
developeithat they can be usedto obtain confidentresultsin nonexperimentaland
weak iuasi-e*perimentalcontexts.As we saw in Chapter 5, researchin the past 2
decadeshas not much supported the notion that a control group can be constructed
'We
ventionsthat areclearlyexogenousshocksto someongoingsystem. arealsoheartenedby the progressbeingmadein the statisticaldomainbecause
it includesprogress
on designconsiderations,
aswell ason analysisper se(e.g.,Rosenbaum,1999a).Ve
areagnosticat this time asto the virtuesof the propensityscoreandinstrumentalvariable approachesthat predominatein discussionsof statisticaladiustmenr.Time will
tell how well
tell
well they
they pan out relative to the results from randomizedexperiments.'We
have surely not heard the last word on this topic.
CONCLUSION
'We
cannot point to one new development that has revolutionized field experimentation in the past few decades,yet we have seena very large number of incremental improvements. As a whole, these improvements allow us to create far better
field experiments than we could do 40 years ago when Campbell and Stanley
(1963) first wrote. In this sense,we are very optimistic about the future. Ve believe
that we will continue to see steadg incremental growth in our knowledge about
how to do better field experiments. The cost of this growth, howeveq is that field
experimentation has become a more specializedtopic, both in terms of knowledge
developmentand of the opportunity to put that knowledge into practice in the conduct of field experiments. As a result, nonspecialistswho wish to do a field experiment may greatly benefit by consulting with those with the expertise,especiallyfor
large experiments, for experiments in which implementation problems may be
high, or for casesin which methodological vulnerabilities will greatly reducecredibility. The same is true, of course, for many other methods. Case-studymethods,
for example, have become highly enough developed that most researcherswould
do an amateurishjob of using them without specializedtraining or supervisedpractice. Such Balkanization of. methodolog)r is, perhaps, inevitable, though none the
lessregrettable.\U7ecan easethe regret somewhat by recognizingthatwith specialization may come faster progress in solving the problems of field experimentation.