Professional Documents
Culture Documents
Estratto - ShadishCookCampbellExperimental2002 2 Bab PDF
Estratto - ShadishCookCampbellExperimental2002 2 Bab PDF
QUASI-EXPERIMENTAL
DESIGNSFORGENERALIZED
ii:.
CAUSALINFERENCE
William R. Shadish
Trru UNIvERSITYop MEvPrrts
** Thomas D. Cook
.jr-*"-
iLli"
NonrrrwpsrERN UNrvPnslrY
'"+.'-, ,
fr
Donald T. Campbell
2002
and
Experiments
Causal
Generalized
lnference
Ex.per'i'ment (ik-spEr'e-mant):[Middle English from Old French from Latin
experimentum, from experiri, to try; seeper- in Indo-European Roots.]
n. Abbr. exp., expt, 1. a. A test under controlled conditions that is
made to demonstratea known truth, examine the validity of a hypothe-
sis, or determine the efficacyof something previously untried' b. The
processof conducting such a test; experimentation. 2' An innovative
"Democracy is only an experiment in gouernment"
act or procedure:
(.V{illiam Ralph lnge).
Cause (k6z): [Middle English from Old French from Latin causa' teason,
purpose.] n. 1. a. The producer of an effect, result, or consequence.
b. The one, such as a person, an event' or a condition, that is responsi-
ble for an action or a result. v. 1. To be the causeof or reason for; re-
sult in. 2. To bring about or compel by authority or force.
AND CAUSATION
EXPERIMENTS
A sensiblediscussionof experimentsrequiresboth a vocabularyfor talking about
causationand an understandingof key conceptsthat underliethat vocabulary.
DefiningCause,Effect,and CausalRelationships
Most peopleintuitively recognizecausalrelationshipsin their daily lives.For in-
stance,you may say that another automobile'shitting yours was a causeof the
damageto your car; that the number of hours you spentstudyingwas a causeof
your testgrades;or that the amountof food a friend eatswas a causeof his weight.
You may evenpoint to more complicatedcausalrelationships,noting that a low
test gradewas demoralizing,which reducedsubsequentstudying,which caused
evenlower grades.Here the samevariable(low grade)can be both a causeand an
effect,and there can be a reciprocal relationship betweentwo variables (low
gradesand not studying)that causeeachother.
Despitethis intuitive familiarity with causalrelationsbips,a precisedefinition
of causeand effecthaseludedphilosophersfor centuries.lIndeed,the definitions
1. Our analysisrefldctsthe useof the word causationin ordinary language,not the more detaileddiscussionsof
causeby philosophers.Readersinterestedin suchdetail may consult a host of works that we referencein this
chapter,includingCook and Campbell(1979).
4 | 1. EXPERTMENTS
AND GENERALTZED
CAUSAL
INFERENCE
Cause
'We
Considerthe causeof a forest fire. know that fires start in differentways-a
match tossedfrom a ca\ a lightning strike, or a smolderingcampfire,for exam-
ple. None of thesecausesis necessarybecausea forest fire can start evenwhen,
say'a match is not present.Also, none of them is sufficientto start the fire. After
all, a match must stay "hot" long enoughto start combustion;it must contact
combustiblematerial suchas dry leaves;theremust be oxygenfor combustionto
occur; and the weather must be dry enoughso that the leavesare dry and the
match is not dousedby rain. So the match is part of a constellationof conditions
without which a fire will not result,althoughsomeof theseconditionscan be usu-
ally takenfor granted,suchasthe availabilityof oxygen.A lightedmatchis, rhere-
fore, what Mackie (1,974)called an inus condition-"an insufficient but non-
redundantpart of an unnecessary but sufficient condition" (p. 62; italicsin orig-
inal). It is insufficientbecausea match cannot start a fire without the other con-
ditions. It is nonredundant only if it adds something fire-promoting that is
uniquelydifferent from what the other factors in the constellation(e.g.,oxygen,
dry leaves)contributeto startinga fire; after all,it would beharderro saywhether
the match causedthe fire if someoneelsesimultaneouslytried startingit with a
cigarettelighter.It is part of a sufficientcondition to start a fire in combination
with the full constellationof factors.But that condition is not necessary because
thereare other setsof conditionsthat can also start fires.
A researchexampleof an inus condition concernsa new potentialtreatment
for cancer.In the late 1990s,a teamof researchers in Bostonheadedby Dr. Judah
Folkman reportedthat a new drug calledEndostatinshrank tumors by limiting
their blood supply (Folkman, 1996).Other respectedresearchers could not repli-
catethe effectevenwhen usingdrugsshippedto them from Folkman'slab. Scien-
tists eventuallyreplicatedthe resultsafter they had traveledto Folkman'slab to
learnhow to properlymanufacture,transport,store,and handlethe drug and how
to inject it in the right location at the right depth and angle.One observerlabeled
thesecontingenciesthe "in-our-hands" phenomenon,meaning "even we don't
AND CAUSATIONI S
EXPERIMENTS
know which details are important, so it might take you some time to work it out"
(Rowe, L999, p.732). Endostatin was an inus condition. It was insufficientcause
by itself, and its effectivenessrequired it to be embedded in a larger set of condi-
tions that were not even fully understood by the original investigators.
Most causesare more accurately called inus conditions. Many factors are usu-
ally required for an effectto occur, but we rarely know all of them and how they
relate to each other. This is one reason that the causal relationships we discussin
this book are not deterministic but only increasethe probability that an effect will
occur (Eells,1,991,;Holland, 1,994).It also explains why a given causalrelation-
ship will occur under some conditions but not universally across time, space,hu-
-"r pop,rlations, or other kinds of treatments and outcomes that are more or less
related io those studied. To different {egrees, all causal relationships are context
dependent,so the generalizationof experimental effects is always at issue.That is
*hy *. return to such generahzationsthroughout this book.
Effect
'We that
can better understand what an effect is through a counterfactual model'l'973'
goes back at least to the 18th-century philosopher David Hume (Lewis,
p. SSel. A counterfactual is something that is contrary to fact. In an experiment,
ie obseruewhat did happez when people received a treatment. The counterfac-
tual is knowledge of what would haue happened to those same people if they si-
multaneously had not receivedtreatment. An effect is the difference betweenwhat
did happen and what would have happened.
'We
cannot actually observe a counterfactual. Consider phenylketonuria
(PKU), a genetically-based metabolic diseasethat causesmental retardation unless
treated during the first few weeks of life. PKU is the absenceof an enzyme that
would otherwise prevent a buildup of phenylalanine, a substance toxic to the
nervous system. Vhen a restricted phenylalanine diet is begun early and main-
tained, reiardation is prevented. In this example, the causecould be thought of as
the underlying genetic defect, as the enzymatic disorder, or as the diet. Each im-
plies a difierenicounterfactual. For example, if we say that a restricted phenyl-
alanine diet causeda decreasein PKU-basedmental retardation in infants who are
phenylketonuric
'h"d at birth, the counterfactual is whatever would have happened
t'h.r. sameinfants not receiveda restricted phenylalanine diet. The samelogic
applies to the genetic or enzymatic version of the cause. But it is impossible for
theseu.ry ,"-i infants simultaneously to both have and not have the diet, the ge-
netic disorder, or the enzyme deficiency.
So a central task for all cause-probing research is to create reasonable ap-
proximations to this physically impossible counterfactual. For instance, if it were
ethical to do so, we might contrast phenylketonuric infants who were given the
diet with other phenylketonuric infants who wer not given the diet but who were
similar in many ways to those who were (e.g., similar face) gender,age, socioeco-
nomic status, health status). Or we might (if it were ethical) contrast infants who
I
6 I 1. EXPERIMENTS
ANDGENERALIZED
CAUSAL
INFERENCE
were not on the diet for the first 3 months of their lives with those same infants
after they were put on the diet starting in the 4th month. Neither of these ap-
proximations is a true counterfactual. In the first case,the individual infants in the
treatment condition are different from those in the comparison condition; in the
second case, the identities are the same, but time has passedand many changes
other than the treatment have occurred to the infants (including permanent dam-
age done by phenylalanine during the first 3 months of life). So two central tasks
in experimental design are creating a high-quality but necessarilyimperfect source
of counterfactual inference and understanding how this source differs from the
treatment condition.
This counterfactual reasoning is fundarnentally qualitative becausecausal in-
ference, even in experiments, is fundamentally qualitative (Campbell, 1975;
Shadish, 1995a; Shadish 6c Cook, 1,999). However, some of these points have
been formalized by statisticiansinto a specialcasethat is sometimescalled Rubin's
"1.974,'1.977,1978,79861.
CausalModel (Holland, 1,986;Rubin, This book is not
about statistics, so we do not describethat model in detail ('West,Biesanz,& Pitts
[2000] do so and relate it to the Campbell tradition). A primary emphasisof Ru-
bin's model is the analysis of causein experiments, and its basic premisesare con-
sistent with those of this book.2 Rubin's model has also been widely used to ana-
lyze causal inference in case-control studies in public health and medicine
(Holland 6c Rubin, 1988), in path analysisin sociology (Holland,1986), and in
a paradox that Lord (1967) introduced into psychology (Holland 6c Rubin,
1983); and it has generatedmany statistical innovations that we cover later in this
book. It is new enough that critiques of it are just now beginning to appear (e.g.,
Dawid, 2000; Pearl, 2000). tUfhat is clear, however, is that Rubin's is a very gen-
eral model with obvious and subtle implications. Both it and the critiques of it are
required material for advanced students and scholars of cause-probingmethods.
CausalRelationship
How do we know if cause and effect are related? In a classic analysis formalized
by the 19th-century philosopher John Stuart Mill, a causal relationship exists if
(1) the causeprecededthe effect, (2) the causewas related to the effect,and (3) we
can find no plausible alternative explanation for the effect other than the cause.
These three characteristics mirror what happens in experiments in which (1) we
manipulate the presumed cause and observe an outcome afterward; (2) we see
whether variation in the cause is related to variation in the effect; and (3) we use
various methods during the experiment to reduce the plausibility of other expla-
nations for the effect, along with ancillary methods to explore the plausibility of
those we cannot rule out (most of this book is about methods for doing this).
2. However, Rubin's model is not intended to say much about the matters of causal generalization that we address
in this book.
EXPERTMENTS | 7
AND CAUSATTON
I
and Confounds
Correlation,
Causation,
A well-known maxim in research is: Correlation does not proue causation. This is
so becausewe may not know which variable came first nor whether alternative ex-
planations for the presumed effectexist. For example, supposeincome and educa-
tion are correlated.Do you have to have a high income before you can aff.ordto pay
for education,or do you first have to get a good education before you can get a bet-
ter paying job? Each possibility may be true, and so both need investigation.But un-
til those investigationsare completed and evaluatedby the scholarly communiry a
simple correlation doesnot indicate which variable came first. Correlations also do
little to rule out alternative explanations for a relationship between two variables
such as education and income. That relationship may not be causal at all but rather
due to a third variable (often called a confound), such as intelligence or family so-
cioeconomicstatus,that causesboth high education and high income. For example,
if high intelligencecausessuccessin education and on the job, then intelligent peo-
ple would have correlatededucation and incomes,not becauseeducation causesin-
come (or vice versa) but becauseboth would be causedby intelligence.Thus a cen-
tral task in the study of experiments is identifying the different kinds of confounds
that can operate in a particular researcharea and understanding the strengthsand
weaknessesassociatedwith various ways of dealing with them
Causes
and Nonmanipulable
Manipulable
In the intuitive understandingof experimentationthat most peoplehave,it makes
senseto say,"Let's seewhat happensif we requirewelfarerecipientsto work"; but
it makesno senseto say,"Let's seewhat happensif I changethis adult maleinto a
three-year-old girl." And so it is alsoin scientificexperiments.Experimentsexplore
the effectsof things that can be manipulated,such as the dose of a medicine,the
amount of a welfarecheck,the kind or amount of psychotherapyor the number
of childrenin a classroom.Nonmanipulableevents(e.g.,the explosionof a super-
nova) or attributes(e.g.,people'sages,their raw geneticmaterial,or their biologi-
cal sex)cannotbe causesin experimentsbecause we cannotdeliberatelyvary them
to seewhat then happens.Consequently, most scientistsand philosophersagree
that it is much harderto discoverthe effectsof nonmanipulablecauses.
I
8 1. EXeERTMENTS cAUsALTNFERENcE
ANDGENERALTzED
|
and CausalExplanation
CausalDescription
The uniquestrengthof experimentationis in describingthe consequences attrib-
utableto deliberatelyvaryinga treatment.'Wecall this causaldescription.In con-
trast, experimentsdo lesswell in clarifying the mechanismsthrough which and
the conditionsunder which that causalrelationshipholds-what we call causal
explanation.For example,most childrenvery quickly learnthe descriptivecausal
relationshipbetweenflicking a light switch and obtainingillumination in a room.
However,few children (or evenadults)can fully explain why that light goeson.
To do so, they would haveto decomposethe treatment(the act of flicking a light
switch)into its causallyefficaciousfeatures(e.g.,closingan insulatedcircuit) and
its nonessentialfeatures(e.g.,whetherthe switch is thrown by hand or a motion
detector).They would haveto do the samefor the effect (eitherincandescentor
fluorescentlight can be produced,but light will still be produced whether the
light fixture is recessedor not). For full explanation,they would then have to
show how the causallyefficaciousparts of the treatmentinfluencethe causally
affectedparts of the outcomethrough identified mediating processes(e.g.,the
I
ANDGENERALIZED
1O I T. CXPTRIMENTS INFERENCE
CAUSAL
passageof electricity through the circuit, the excitation of photons).3 ClearlS the
causeof the light going on is a complex cluster of many factors. For those philoso-
phers who equate cause with identifying that constellation of variables that nec-
essarily inevitably and infallibly results in the effect (Beauchamp,1.974),talk of
cause is not warranted until everything of relevanceis known. For them, there is
no causal description without causal explanation. Whatever the philosophic mer-
its of their position, though, it is not practical to expect much current social sci-
ence to achieve such complete explanation.
The practical importance of causal explanation is brought home when the
switch fails to make the light go on and when replacing the light bulb (another
easily learned manipulation) fails to solva the problem. Explanatory knowledge
then offers clues about how to fix the problem-for example, by detecting and re-
pairing a short circuit. Or if we wanted to create illumination in a place without
lights and we had explanatory knowledge, we would know exactly which features
of the cause-and-effectrelationship are essentialto create light and which are ir-
relevant. Our explanation might tell us that there must be a source of electricity
but that that source could take several different molar forms, such as abattery, a
generator, a windmill, or a solar array. There must also be a switch mechanism to
close a circuit, but this could also take many forms, including the touching of two
bare wires or even a motion detector that trips the switch when someone enters
the room. So causal explanation is an important route to the generalization of
causal descriptions becauseit tells us which features of the causal relationship are
essentialto transfer to other situations.
This benefit of causal explanation helps elucidate its priority and prestige in
all sciencesand helps explain why, once a novel and important causal relationship
is discovered, the bulk of basic scientific effort turns toward explaining why and
how it happens. Usuallg this involves decomposing the causeinto its causally ef-
fective parts, decomposing the effects into its causally affected parts, and identi-
fying the processesthrough which the effective causal parts influence the causally
affected outcome parts.
These examplesalso show the close parallel between descriptive and explana-
tory causation and molar and molecular causation.aDescriptive causation usually
concerns simple bivariate relationships between molar treatments and molar out-
comes, molar here referring to a package that consistsof many different parts. For
instance, we may find that psychotherapy decreasesdepression,a simple descrip-
tive causal relationship benveen a molar treatment package and a molar outcome.
However, psychotherapy consists of such parts as verbal interactions, placebo-
3. However, the full explanationa physicistwould offer might be quite different from this electrician's
explanation,perhapsinvoking the behaviorof subparticles.This differenceindicatesiust how complicatedis the
notion of explanationand how it can quickly becomequite complex once one shifts levelsof analysis.
4. By molar, we mean somethingtaken as a whole rather than in parts. An analogyis to physics,in which molar
might refer to the propertiesor motions of masses,as distinguishedfrom those of moleculesor atomsthat make up
thosemasses.
AND CAUSATIONI 11
EXPERIMENTS
I
MODERNDESCRIPTIONS
OF EXPERIMENTS
Some of the terms used in describing modern experimentation (seeTable L.L) are
unique, clearly defined, and consistently used; others are blurred and inconsis-
tently used. The common attribute in all experiments is control of treatment
(though control can take many different forms). So Mosteller (1990, p. 225)
writes, "fn an experiment the investigator controls the application of the treat-
ment"l and Yaremko, Harari, Harrison, and Lynn (1,986,p.72) write, "one or
more independent variables are manipulated to observe their effects on one or
more dependentvariables." However, over time many different experimental sub-
types have developed in responseto the needs and histories of different sciences
'Winston
('Winston, 1990; 6c Blais, 1.996\.
TABLE1.1TheVocabularyof Experiments
Experiment:
A studyin whichan intervention
is deliberately to observe
introduced itseffects.
RandomizedExperiment:
An experiment in whichunitsareassignedto receivethe treatmentor
conditionby a randomprocess
an alternative suchasthe tossof a coinor a tableof
randomnumbers.
An experiment
Quasi-Experiment: to conditions
in whichunitsarenot assigned randomly.
NaturalExperiment:
Not reallyan experiment
because the causeusuallycannotbe
manipulated;
a studythat contrasts
a naturally
occurringeventsuchasan earthquake
with
a comoarisoncondition.
Correlational
Study:Usuallysynonymouswith nonexperimentalor observational
study;a study
thatsimplyobserves
the sizeanddirection amongvariables.
of a relationship
I
I tr
OF EXPERIMENTS
MODERNDESCRIPTIONS
Experiment
Randomized
The most clearlydescribedvariant is the randomizedexperiment,widely credited
to Sir RonaldFisher(1,925,1926).Itwas first usedin agriculturebut laterspread
to other topic areasbecauseit promisedcontrol over extraneoussourcesof vari-
ation without requiringthe physicalisolationof the laboratory.Its distinguishing
featureis clear and important-that the varioustreatmentsbeingcontrasted(in-
cludingno treatmentat all) are assignedto experimentalunits' by chance,for ex-
ample,by cointossor useof a table of random numbers.If implementedcorrectlS
,"rdo- assignmentcreatestwo or more groupsof units that are probabilistically
similarto .".h other on the average.6Hence,any outcomedifferencesthat are ob-
servedbetweenthosegroupsat the end,ofa study arelikely to be dueto treatment'
not to differencesbetweenthe groupsthat alreadyexistedat the start of the study.
Further,when certainassumptionsare met, the randomizedexperimentyieldsan
estimateof the sizeof a treatmenteffectthat has desirablestatisticalproperties'
along with estimatesof the probability that the true effectfalls within a defined
confidenceinterval.Thesefeaturesof experimentsare so highly prized that in a
researchareasuchas medicinethe randomizedexperimentis often referredto as
the gold standardfor treatmentoutcomeresearch.'
Closelyrelatedto the randomizedexperimentis a more ambiguousand in-
consistentlyusedterm, true experiment.Someauthorsuseit synonymouslywith
randomizedexperiment(Rosenthal& Rosnow,1991').Others useit more gener-
ally to refer to any studyin which an independentvariableis deliberately manip-
'We
ulated (Yaremkoet al., 1,9861anda dependentvariableis assessed. shall not
usethe term at all givenits ambiguity and given that the modifier true seemsto
imply restrictedclaimsto a singlecorrectexperimentalmethod.
Quasi-Experiment
Much of this book focuseson a class of designsthat Campbell and Stanley
(1,963)popularizedasquasi-experiments.s sharewith all other
Quasi-experiments
14 I 1. EXPERIMENTS
AND GENERALIZED
CAUSAL
INFERENCE
I
or others decide which persons should get which treatment. Howeveq researchers
who use quasi-experimentsmay still have considerablecontrol over selectingand rl
schedulingmeasures,over how nonrandom assignmentis executed,over the kinds
of comparison groups with which treatment,groups are compared, and over some
aspectsof how treatment is scheduled.As Campbell and Stanleynote:
II tt
OFEXPERIMENTS
MODERNDESCRTPTONS
NaturalExperiment
The term natural experiment describesa naturally-occurring contrast between a
treatment and a comparisoncondition (Fagan, 1990; Meyer, 1995;Zeisel,1,973l.
Often the treatments are not even potentially manipulable, as when researchers
retrospectivelyexamined whether earthquakesin California causeddrops in prop-
erty values (Brunette, 1.995; Murdoch, Singh, 6c Thayer, 1993). Yet plausible
causal inferences about the effects of earthquakes are easy to construct and de-
fend. After all, the earthquakesoccurred before the observations on property val-
ues,and it is easyto seewhether earthquakesare related to properfy values. A use-
ful source of counterfactual inference can be constructed by examining property
values in the same locale before the earthquake or by studying similar localesthat
did not experience an earthquake during the bame time. If property values
dropped right after the earthquake in the earthquake condition but not in the com-
parison condition, it is difficult to find an alternative explanation for that drop.
Natural experiments have recently gained a high profile in economics. Before
the 1990s economists had great faith in their ability to produce valid causal in-
ferencesthrough statistical adjustments for initial nonequivalence between treat-
ment and control groups. But two studies on the effects of job training programs
showed that those adjustments produced estimates that were not close to those
generated from a randomized experiment and were unstable across tests of the
model's sensitivity (Fraker 6c Maynard, 1,987; Lalonde, 1986). Hence, in their
searchfor alternative methods, many economistscame to do natural experiments,
such as the economic study of the effects that occurred in the Miami job market
when many prisoners were releasedfrom Cuban jails and allowed to come to the
United States(Card, 1990). They assumethat the releaseof prisoners (or the tim-
ing of an earthquake) is independent of the ongoing processesthat usually affect
unemployment rates (or housing values). Later we explore the validity of this
assumption-of its desirability there can be little question.
18 I 1. EXPERIMENTS
AND GENERALIZED INFERENCE
CAUSAL
Nonexperimental
Designs
The termscorrelationaldesign,passiveobservationaldesign,and nonexperimental
designrefer to situationsin which a presumedcauseand effect are identified and
measuredbut in which other structural featuresof experimentsare missing.Ran-
dom assignmentis not part of the design,nor are suchdesignelementsas pretests
and control groupsfrom which researchers might constructa usefulcounterfactual
inference.Instead,relianceis placedon measuringalternativeexplanationsindi-
vidually and then statisticallycontrolling for them. In cross-sectionalstudiesin
which all the data aregatheredon the respondentsat one time, the researchermay
not even know if the causeprecedesthe dffect. When thesestudiesare used for
causalpurposes,the missingdesignfeaturescan be problematicunlessmuch is al-
ready known about which alternativeinterpretationsare plausible,unlessthose
that are plausiblecan be validly measured,and unlessthe substantivemodel used
for statisticaladjustmentis well-specified.
Theseare difficult conditionsto meetin
the real world of researchpractice,and thereforemany commentatorsdoubt the
potentialof suchdesignsto supportstrongcausalinferencesin most cases.
EXPERIMENTS
ANDTHEGENERALIZATION
OF
CAUSALCONNECTIONS
The strength of experimentation is its ability to illuminate causal inference. The
weaknessof experimentation is doubt about the extent to which that causal rela-
'We
tionship generalizes. hope that an innovative feature of this book is its focus
on generalization. Here we introduce the general issuesthat are expanded in later
chapters.
Most Experiments
Are HighlyLocalBut Have
GeneralAspirations
S,
9. We oversimplify Cronbach'spresentationhere for pedagogicalreasons.For example,Cronbach only usedcapital
not small s, so that his system,eferred only to ,tos, not utos. He offered diverseand not always consistentdefinitions
do here.
of UTOS and *UTOS, in particular. And he doesnot usethe word generalizationin the samebroad way we
I
20 I 1. EXPERIMENTS
AND GENERALIZED INFERENCE
CAUSAL
ConstructValidity:CausalGeneralization
as Representation
The first causal generalization problem concerns how to go from the particular
units, treatments, observations, and settings on which data are collected to the
higher order constructs these instancesrepresent.These constructs are almost al-
ways couched in terms that are more abstract than the particular instancessam-
pled in an experiment. The labels may pertain to the individual elementsof the ex-
periment (e.g., is the outcome measured by a given test best described as
intelligence or as achievement?).Or the labels may pertain to the nature of rela-
tionships among elements, including causal relationships, as when cancer treat-
ments are classified as cytotoxic or cytostatic depending on whether they kill tu-
mor cells directly or delay tumor growth by modulating their environment.
Consider a randomized experiment by Fortin and Kirouac (1.9761.The treatment
was a brief educational course administered by severalnurses,who gave a tour of
their hospital and covered some basic facts about surgery with individuals who
were to have elective abdominal or thoracic surgery 1-5to 20 days later in a sin-
gle Montreal hospital. Ten specific outcome measureswere used after the surgery,
such as an activities of daily living scaleand a count of the analgesicsused to con-
trol pain. Now compare this study with its likely t^rget constructs-whether
patient education (the target cause)promotes physical recovery (the targt effect)
"*ong surgical patients (the target population of units) in hospitals (the target
univeise ofiettings). Another example occurs in basic research,in which the ques-
tion frequently aiises as to whether the actual manipulations and measuresused
in an experiment really tap into the specific cause and effect constructs specified
by the theory. One way to dismiss an empirical challenge to a theory is simply to
make the casethat the data do not really represent the concepts as they are spec-
ified in the theory.
Empirical resnlts often force researchersto change their initial understanding
of whaithe domain under study is. Sometimesthe reconceptuahzation leads to a
more restricted inference about what has been studied. Thus the planned causal
agent in the Fortin and Kirouac (I976),study-patie,nt education-might need to
b! respecified as informational patient education if the information component of
the treatment proved to be causally related to recovery from surgery but the tour
of the hospital did not. Conversely data can sometimes lead researchersto think
in terms o?,"rg., constructs and categoriesthat are more general than those with
which they began a researchprogram. Thus the creative analyst of patient educa-
tion studies mlght surmise that the treatment is a subclass of interventions that
"perceived control" or that recovery from surgery can be
function by increasing
;'p.tronal coping." Subsequentreaders of the study can
treated as a subclas of
even add their own interpietations, perhaps claiming that perceived control is re-
ally just a special caseof the even more general self-efficacy construct. There is a
sobtie interplay over time among the original categories the researcherintended
to represeni, the study as it was actually conducted, the study results, and subse-
qrr..ri interpretations. This interplay can change the researcher'sthinking about
what the siudy particulars actually achieved at a more conceptual level, as can
feedback fromreaders. But whatever reconceptualizationsoccur' the first problem
of causal generaltzationis always the same: How can we generalizefrom a sam-
ple of instancesand the data patterns associatedwith them to the particular tar-
get constructs they represent?
as Extrapolation
ExternalValidity:CausalGeneralization
The secondproblem of generalizationis to infer whether a causalrelationship
holdsovervariationsin p.rrorrt, settings,treatments,and outcomes.For example,
someonereadingthe resultsof an experimenton the effectsof a kindergarten
Head Startprogiam on the subsequent grammarschoolreadingtestscoresof poor
African Americanchildrenin Memphis during the 1980smay want to know if a
programwith partially overlappingcognitiveand socialdevelopmentgoals_would
be aseffectivein improvingthi mathematicstest scoresof poor Hispanicchildren
in Dallas if this programwere to be implementedtomorrow.
This exampl. again reminds us that generahzationis not a synonym for
broader applicatiorr.H.r., generahzationis from one city to another city and
1. EXPERIMENTS
AND GENERALIZED INFERENCE
CAUSAL
Approaches
to MakingCausalGeneralizations
\Thichever way the causal generalization issue is framed, experiments do not
seem at first glance to be very useful. Almost invariablS a given experiment uses
a limited set of operations to represent units, treatments, outcomes, and settings.
This high degree of localization is not unique to the experiment; it also charac-
terizes case studies, performance monitoring systems, and opportunistically-
administered marketing questionnaires given to, say, a haphazard sample of re-
spondents at local shopping centers (Shadish, 1995b). Even when questionnaires
are administered to nationally representative samples, they are ideal for repre-
senting that particular population of persons but have little relevanceto citizens
outside of that nation. Moreover, responsesmay also vary by the setting in which
the interview took place (a doorstep, a living room, or a work site), by the time
of day at which it was administered, by how each question was framed, or by the
particular race, age,and gender combination of interviewers. But the fact that the
experiment is not alone in its vulnerability to generalization issuesdoes not make
it any less a problem. So what is it that justifies any belief that an experiment can
achieve a better fit between the sampling particulars of a study and more general
inferences to constructs or over variations in persons, settings, treatments, and
outcomes?
oF cAUsALcoNNEcrtoNs I tt
ANDTHEGENERALtzATtoN
EXeERTMENTs
Samplingand CausalGeneralization
The methodmost often recommendedfor achievingthis closefit is the useof for-
mal probabiliry samplingof instancesof units, treatments,observations,or set-
tings (Rossi,Vlright, & Anderson,L983). This presupposes that we have clearly
deiineatedpopulationsof eachand that we can samplewith known probability
from within eachof thesepopulations.In effect,this entailsthe random selection
of instances,to be carefullydistinguishedfrom random assignmentdiscussed ear-
lier in this chapter.Randomselectioninvolvesselectingcases by chance to repre-
sentthat popuiation,whereasrandom assignmentinvolvesassigningcasesto mul-
tiple conditions.
In cause-probingresearchthat is not experimental,random samplesof indi-
viduals"r. oft.n nr.d. Large-scale longitudinalsurveyssuchasthe PanelStudyof
IncomeDynamicsor the National Longitudinal Surveyare usedto representthe
populationof the United States-or certainagebracketswithin it-and measures
Lf pot.ntial causesand effectsare then relatedto each other using time lags in
,nr^"r.rr.-ent and statisticalcontrolsfor group nonequivalence. All this is donein
hopesof approximatingwhat a randomizedexperimentachieves.However,cases
of random ielection from a broad population followed by random assignment
from within this population are much rarer (seeChapter 12 for examples).Also
rare arestudiesoi t".rdotn selectionfollowed by a quality quasi-experiment. Such
experimentsrequirea high levelof resourcesand a degreeof logisticalcontrol that
is iarely feasible,so many researchers prefer to rely on an implicit set of nonsta-
tistical heuristicsfor generalizationthat we hope to make more explicit and sys-
tematicin this book.
Random selectionoccurseven more rarely with treatments'outcomes,and
settingsthan with people.Considerthe outcomesobservedin an experiment.How
ofterrlre they raniomly sampled?'Wegrant that the domain samplingmodel of
classicaltestiheory (Nunnally 6c Bernstein,1994)assumesthat the itemsusedto
measurea constructhavebeenrandomly sampledfrom a domain of all possible
items. However,in actual experimentalpracticefew researchersever randomly
sampleitemswhen constructingmeasures.Nor do they do so when choosingma-
nipulationsor settings.For instance,many settingswill not agreeto be sampled,
"rid ,o1n. of the settingsthat agreeto be randomly sampledwill almostcertainly
not agreeto be randomlyassignedto conditions.For treatments,no definitivelist
of poisible treatmentsusuallyexists,as is most obvious in areasin which treat-
-*,, are being discoveredand developedrapidly, such as in AIDS research.In
general,then, random samplingis alwaysdesirable,but it is only rarely and con-
tingently feasible.
"However,
formal samplingmethodsare not the only option. Two informal, pur-
posive samplingmethodrare sometimesuseful-purposive sampling of heteroge-
neousinstancesand purposivesamplingof typical instances.In the former case'the
aim is to includeinrLni.r chosendeliberatelyto reflect diversity on presumptively
important dimensions,eventhough the sampleis not formally random. In the latter
24 I .l. TxpEnIMENTS INFERENCE
CAUSAL
ANDGENERALIZED
case,the aim is to explicate the kinds of units, treatments, observations, and settings
to which one most wants to generalize andthen to selectat least one instance of each
class that is impressionistically similar to the class mode. Although these purposive
sampling methods are more practical than formal probability sampling, they are not
backed by a statistical logic that justifies formal generalizations.Nonetheless, they
are probabty the most commonly used of all sampling methods for facilitating gen-
eralizations. A task we set ourselvesin this book is to explicate such methods and to
describe how they can be used more often than is the casetoday.
However, sampling methods of any kind are insufficient to solve either prob-
lem of generalization. Formal probability sampling requires specifying a target
population from which sampling then takes place, but defining such populations
is difficult for some targets of generalization such as treatments. Purposive sam-
pling of heterogeneousinstancesis differentially feasible for different elementsin
a study; it is often more feasible to make measuresdiverse than it is to obtain di-
verse settings, for example. Purposive sampling of typical instancesis often feasi-
ble when target modes, medians, or means are known, but it leaves questions
about generalizationsto a wider range than is typical. Besides,as Cronbach points
out, most challenges to the causal generalization of an experiment typically
emerge after a study is done. In such cases,sampling is relevant only if the in-
stancesin the original study were sampled diversely enough to promote responsi-
ble reanalysesof the data to seeif a treatment effect holds acrossmost or all of the
targets about which generahzation has been challenged. But packing so many
sourcesof variation into a single experimental study is rarely practical and will al-
most certainly conflict with other goals of the experiment. Formal sampling meth-
ods usually offer only a limited solution to causal generalizationproblems. A the-
ory of generalizedcausal inference needsadditional tools.
A GroundedTheoryof CausalGeneralization
Practicingscientistsroutinely make causal generalizations in their research,and
they almostneveruseformal probability samplingwhen they do. In this book, we
presenta theory of causalgeneralization that is groundedin the actualpracticeof
science(Matt, Cook, 6c Shadish,2000). Although this theory was originally de-
velopedfrom ideasthat were groundedin the constructand externalvalidiry lit-
eratures(Cook, 1990,1991.),we havesincefound that theseideasarecommonin
a diverseliteratureabout scientificgeneralizations (e.g.,Abelson,1995;Campbell
& Fiske, 1.959;Cronbach& Meehl, 1955; Davis, 1994; Locke, 1'986;Medin,
1989;Messick,1ggg,1'995;Rubins,1.994;'Willner, Hayward,Tu-
1,991';'$7ilson,
\7e providemore details
nis, Bass,& Guyatt, 1,995];t. about this grounded theory
in Chapters1L through L3, but in brief it suggests that scientistsmakecausalgen-
eralizationsin their work by usingfive closelyrelatedprinciples:
"L. the apparentsimilaritiesbetweenstudy opera-
SurfaceSimilarity.They assess
tions and the prototypicalcharacteristicsof the target of generalization.
I
AND THEGENERALIZATION
EXPERIMENTS II ZS
OFCAUSALCONNECTIONS
2. Ruling Out lrreleuancies.They identify those things that are irrelevant because
they do not change a generalization.
3. Making Discriminations. They clarify k.y discriminations that limit
generalization.
4. Interpolation and Extrapolation. They make interpolations to unsampled val-
ues within the range of the sampled instances and, much more difficult, they
explore extrapolations beyond the sampled range.
5 . Causal Explanation. They develop and test explanatory theories about the pat-
tern of effects,causes,and mediational processesthat are essentialto the trans-
fer of a causalrelationship.
In this book, we want to show how scientistscan and do use thesefive princi-
ples to draw generalizedconclusions dbout a causal connection. Sometimes the
conclusion is about the higher order constructs to use in describing an obtained
connection at the samplelevel. In this sense,thesefive principles have analoguesor
parallels both in the construct validity literature (e.g.,with construct content, with
loru.rg.nt and discriminant validity, and with the need for theoretical rationales
for consrructs) and in the cognitive scienceand philosophy literatures that study
how people decidewhether instancesfall into a category(e.g.,concerning the roles
that protorypical characteristicsand surface versus deep similarity play in deter-
mining category membership). But at other times, the conclusion about general-
ization refers to whether a connection holds broadly or narrowly over variations
in persons, settings,treatments, or outcomes. Here, too, the principles have ana-
logues or parallels that we can recognizefrom scientific theory and practice, as in
the study of dose-responserelationships (a form of interpolation-extrapolation) or
the appeal to explanatory mechanismsin generalizing from animals to humans (a
form of causal explanation).
Scientistsuse rhese five principles almost constantly during all phases of re-
search.For example, when they read a published study and wonder if some varia-
tion on the study's particulars would work in their lab, they think about similari-
'$7hen
ties of the published study to what they propose to do. they conceptualize
the new study, they anticipate how the instances they plan to study will match the
prototypical featuresof the constructs about which they are curious. They may de-
iign their study on the assumptionthat certain variations will be irrelevant to it but
that others will point to key discriminations over which the causal relationship
does not hold or the very character of the constructs changes.They may include
measuresof key theoretical mechanisms to clarify how the intervention works.
During data analysis, they test all these hypotheses and adjust their construct de-
scriptions to match better what the data suggest happened in the study. The intro-
duction section of their articles tries to convince the reader that the study bears on
specific constructs, and the discussion sometimes speculatesabout how results
-igttt extrapolate to different units, treatments, outcomes, and settings.
Further, practicing scientistsdo all this not just with single studies that they
read or conduct but also with multiple studies. They nearly always think about
26 1. EXPERTMENTS
ANDGENERALIZED INFERENCE
CAUSAL
|
how their own studiesfit into a larger literature about both the constructsbeing
measuredand the variablesthat may or may not bound or explain a causalconnec-
tion, often documentingthis fit in the introduction to their study.And they apply all
five principleswhen they conduct reviewsof the literature,in which they make in-
ferencesabout the kinds of generalizations that a body of researchcan suppoft.
Throughoutthis book, and especiallyin Chapters11 to L3, we providemore
detailsabout this groundedtheory of causal generalizationand about the scientific
Adopting this groundedtheoryof generalization
practicesthat it suggests. doesnot
imply a rejectionof formal probabilitysampling.Indeed, we recommendsuchsam-
pling unambiguouslywhenit is feasible,alongwith purposive samplingschemes to
aid generalization when formal randomselectionmethodscannotbe implemented.
But we alsoshow that samplingis just one methodthat practicingscientistsuseto
make causalgeneralizations, along with practicallogic, applicationof diversesta-
tistical methods,and useof featuresof designother than sampling.
AND METASCIENCE
EXPERIMENTS
Extensivephilosophicaldebatesometimessurroundsexperimentation.Here we
briefly summarizesomekey featuresof thesedebates,and then we discusssome
implications of thesedebatesfor experimentation.However,there is a sensein
which all this philosophicaldebateis incidentalto the practiceof experimentation.
Experimentationis as old as humanity itself, so it precededhumanity'sphilo-
sophicaleffortsto understandcausationand genenlizationby thousandsof years.
Even over just the past 400 yearsof scientificexperimentation,we can seesome
constancyof experimentalconcept and method, whereasdiversephilosophical
"Ex-
conceptions of the experimenthavecomeand gone.As Hacking(1983)said,
perimentationhas a life of its own" (p. 150). It has beenone of science's most
powerful methodsfor discoveringdescriptivecausalrelationships,and it hasdone
so well in so many ways that its placein scienceis probably assuredforever.To
justify its practicetodag a scientistneednot resortto sophisticatedphilosophical
reasoningabout experimentation.
Nonetheless,it doeshelp scientiststo understandthesephilosophicaldebates.
For example,previousdistinctionsin this chapterbetweenmolar and molecular
causation,descriptiveand explanatorycause,or probabilisticand deterministic
causalinferencesall help both philosophersand scientiststo understandbetter
both the purposeand the resultsof experiments(e.g.,Bunge,1959; Eells, 1991';
Hart & Honore, 1985;Humphreys,"t989;Mackie, 1'974;Salmon,7984,1989;
Sobel,1993;P.A. \X/hite,1990).Here we focus on a differentand broadersetof
critiquesof scienceitself,not only from philosophybut alsofrom the history,so-
ciologS and psychologyof science(seeusefulgeneralreviewsby Bechtel,1988;
H. I. Brown, 1977; Oldroyd, 19861.Someof theseworks have beenexplicitly
about the nature of experimentation,seekingto createa justified role for it (e.g.,
I
AND METASCIENCE
EXPERIMENTS I 27
'1.990;
Bhaskar,L975;Campbell,1982,,1988;Danziger, S. Drake, l98l; Gergen,
1,973;Gholson,Shadish, Neimeyer,6d Houts, L989; Gooding,Pinch,6cSchaffer,
'Woolgar,
1,989b;Greenwood, L989; Hacking, L983; Latour, 1'987;Latour 6c
1.979;Morawski, 1988;Orne,1.962;R.RosenthaL,1.966;Shadish & Fuller,L994;
Shapin,1,9941. Thesecritiqueshelp scientiststo seesomelimits of experimenta-
tion in both scienceand society.
TheKuhnianCritique
Kuhn (1962\ describedscientificrevolutionsas differentand partly incommensu-
rableparadigmsthat abruptly succeedgd eachother in time and in which the grad-
ual accumulation of scientificknowledgewas a chimera.Hanson(1958),Polanyi
(1958),Popper('J.959), Toulmin (1'961),Feyerabend(L975),and Quine (1'95t'
1,969)contributedto the critical momentum,in part by exposingthe grossmis-
takesin logicalpositivism'sattemptto build a philosophyof sciencebasedon re-
constructinga successfulsciencesuch as physics.All thesecritiquesdeniedany
firm foundationsfor scientificknowledge(so, by extension,experimentsdo not
provide firm causalknowledge).The logicalpositivistshopedto achievefounda-
tions on which to build knowledgeby tying all theory tightly to theory-freeob-
servationthrough predicatelogic. But this left out important scientificconcepts
that could not be tied tightly to observation;and it failed to recognizethat all ob-
servationsare impregnatedwith substantiveand methodologicaltheory,making
it impossibleto conducttheory-freetests.lt
The impossibility of theory-neutral observation (often referred to as the
Quine-Duhemthesis)impliesthat the resultsof any singletest (and so any single
experiment)are inevitably ambiguous.They could be disputed,for example,on
groundsthat the theoreticalassumptionsbuilt into the outcome measurewere
wrong or that the study made a fatity assumptionabout how high a treatment
dosewas requiredto be effective.Someof theseassumptionsare small,easilyde-
tected,and correctable,suchaswhen a voltmetergivesthe wrong readingbecause
the impedanceof the voltagesourcewas much higherthan that of the meter ('$fil-
son, L952).But other assumptionsare more paradigmlike,impregnatinga theory
so completelythat other parts of the theory makeno sensewithout them (e.g.,the
assumptionthat the earthis the centerof the universein pre-Galileanastronomy).
Becausethe number of assumptionsinvolved in any scientifictest is very large,
researcherscan easily find some assumptionsto fault or can even posit new
Critiques
ModernSocialPsychological
Sociologists working within traditionsvariouslycalledsocialconstructivism,epis-
temologicalrelativism,and the strongprogram(e.g.,Barnes,1974;Bloor,1976;
Collins,l98l;Knorr-Cetina,L981-;Latour 6c'Woolgar,1.979;Mulkay, 1'979)have
shown thoseextrascientificprocesses at work in science.Their empiricalstudies
show that scientistsoften fail to adhereto norms commonlyproposedas part of
good science(e.g.,objectivity neutrality,sharingof information).They havealso
rho*n how that which comesto be reportedas scientificknowledgeis partly de-
terminedby socialand psychologicalforcesand partly by issuesof economicand
political power both within scienceand in the largersociety-issuesthat arerarely
mention;d in publishedresearchreports.The most extremeamongthesesociolo-
gistsattributesall scientificknowledgeto suchextrascientificprocesses, claiming
ihat "the natural world has a small or nonexistentrole in the constructionof sci-
"l'98I,
entificknowledge"(Collins, p. 3).
Collins doesnot denyontologicalrea.lism,that real entitiesexistin the world.
Rather,he deniesepistemological(scientific)realism, that whateverexternal real-
ity may existcanconstrainour scientifictheories.For example,if atomsreally ex-
ist, do they affectour scientifictheoriesat all? If our theory postulatesan atom, is
it describing a realentitythat existsroughly aswe describeit? Epistetnologi,cal rel-
atiuistssuch as Collins respondnegativelyto both questions,believingthat the
most important influencesin scienceare social,psychological,economic,and po-
litical, "ttd th"t thesemight evenbe the only influenceson scientifictheories-This
view is not widely endorsedoutsidea small group of sociologists,but it is a use-
ful counterweightto naiveassumptionsthat scientificstudiessomehowdirectlyre-
veal natur. to r.r,(an assumptiorwe callnaiuerealism).The resultsof all studies,
including experiments,are profoundly subjectto theseextrascientificinfluences,
from their conceptionto reportsof their results.
and Trust
Science
A standard image of the scientist is as a skeptic, a person who only trusts results that
have been personally verified. Indeed, the scientific revolution of the'l'7th century
I
I 29
AND METASCIENCE
EXPERIMENTS
I
claimed that trust, particularly trust in authority and dogma, was antithetical to
good science.Every authoritative assertion,every dogma, was to be open to ques-
tion, and the job of sciencewas to do that questioning.
That image is partly wrong. Any single scientific study is an exercisein trust
(Pinch, 1986; Shapin, 1,994).Studies trust the vast majority of already developed
methods, findings, and concepts that they use when they test a new hypothesis.
For example, statistical theories and methods are usually taken on faith rather
than personally verified, as are measurement instruments. The ratio of trust to
skepticism in any given study is more llke 99% trust to 1% skepticism than the
opposite. Even in lifelong programs of research, the single scientist trusts much
-or. than he or she ever doubts. Indeed, thoroughgoing skepticism is probably
impossible for the individual scientist, po iudge from what we know of the psy-
chology of science(Gholson et al., L989; Shadish 6c Fuller, 1'9941.Finall5 skepti-
cism is not even an accuratecharacterrzation of past scientific revolutions; Shapin
"gentlemanly trust" in L7th-century England was
(1,994) shows that the role of
central to the establishment of experimental science.Trust pervades science,de-
spite its rhetoric of skepticism.
for Experiments
lmplications
The net result of thesecriticismsis a greaterappreciationfor the equivocalityof
all scientificknowledge.The experimentis not a clearwindow that revealsnature
directly to us.To the contrary,experimentsyield hypotheticaland fallible knowl-
edgethat is often dependenton context and imbuedwith many unstatedtheoret-
ical assumprions.Consequentlyexperimentalresultsare partly relativeto those
assumptionsand contextsand might well changewith new assumptionsor con-
constructivistsand relativists.
texts.In this sense,all scientistsare epistemological
The differenceis whether they are strong or weak relativists.Strong relativists
share Collins'sposition that only extrascientificfactors influenceour theories.
'Weak
relativistsbelievethat both the ontologicalworld and the worlds of ideol-
og5 interests,values,hopes,and wishesplay a role in the constructionof scien-
tiiic knowledge.Most practicingscientists,including ourselves,would probably
describethemselves ", Lrrtologicalrealistsbut weak epistemologicalrelativists.l2
To the extent that experimentsrevealnature to us, it is through a very clouded
windowpane(Campbell,1988).
Suchcounterweights to naiveviewsof experimentswere badly needed.As re-
centlyas 30 yearsago,the centralrole of the experimentin sciencewas probably
1.2. If spacepermitred,we could exrendthis discussionto a host of other philosophicalissuesthat have beenraised
about the experiment, such as its role in discovery versusconfirmation, incorrect assertionsthat the experiment is
tied to somespecificphilosophysuch as logical positivismor pragmatism,and the various mistakesthat are
frequentlymadei., suchdiscussions(e.g.,Campbell, 1982,1988; Cook, 1991; Cook 6< Campbell, 1985; Shadish,
1.995a\.
I
30 | 1. EXPERTMENTS
AND GENERALTZED INFERENCE
CAUSAL
taken more for granted than is the case today. For example, Campbell and Stan-
ley (1.9631
describedthemselvesas:
committed to the experiment: as the only means for settling disputes regarding educa-
tional practice, as the only way of verifying educational improvements, and as the only
way of establishing a cumulative tradition in which improvements can be introduced
without the danger of a faddish discard of old wisdom in favor of inferior novelties. (p. 2)
"'experimental method' usedto be
Indeed,Hacking (1983) points out that iust an-
other name for scientific method" (p.149); and experimentation was then a more
fertile ground for examples illustrating basic philosophical issuesthan it was a
source of contention itself. ,
'We
Not so today. now understand better that the experiment is a profoundly
human endeavor,affected by all the same human foibles as any other human en-
deavor, though with well-developed procedures for partial control of some of the
limitations that have been identified to date. Some of these limitations are com-
mon to all science,of course. For example, scientiststend to notice evidencethat
confirms their preferred hypothesesand to overlook contradictory evidence.They
make routine cognitive errors of judgment and have limited capacity to process
large amounts of information. They react to peer pressuresto agreewith accepted
dogma and to social role pressuresin their relationships to students,participants,
and other scientists.They are partly motivated by sociological and economic re-
wards for their work (sadl5 sometimesto the point of fraud), and they display all-
too-human psychological needs and irrationalities about their work. Other limi-
tations have unique relevance to experimentation. For example, if causal results
are ambiguous, as in many weaker quasi-experiments,experimentersmay attrib-
ute causation or causal generalization based on study features that have little to
do with orthodox logic or method. They may fail to pursue all the alternative
causal explanations becauseof a lack of energS a need to achieveclosure, or a bias
toward accepting evidence that confirms their preferred hypothesis.Each experi-
ment is also a social situation, full of social roles (e.g., participant, experimenter,
assistant) and social expectations (e.g., that people should provide true informa-
tion) but with a uniqueness (e.g., that the experimenter does not always tell the
truth) that can lead to problems when social cues are misread or deliberately
thwarted by either party. Fortunately these limits are not insurmountable, as for-
mal training can help overcome some of them (Lehman, Lempert, & Nisbett,
1988). Still, the relationship between scientific results and the world that science
studies is neither simple nor fully trustworthy.
These social and psychological analyseshave taken some of the luster from
the experiment as a centerpieceof science.The experiment may have a life of its
own, but it is no longer life on a pedestal. Among scientists,belief in the experi-
ment as the only meansto settle disputes about causation is gone, though it is still
the preferred method in many circumstances. Gone, too, is the belief that the
power experimental methods often displayed in the laboratory would transfer eas-
ily to applications in field settings. As a result of highly publicized science-related
I
OR CAUSES?I gT
A WORLDWITHOUTEXPERIMENTS
I
ORCAUSES?
A WORLDWITHOUTEXPERIMENTS
To borrow a thought experimentfrom Maclntyre (1981),imaginethat the slates
of scienceand philosophywerewiped cleanand that we had to constructour un-
derstandingof the world anew.As part of that reconstruction,would we reinvent
the notion of a manipulablecause?\7e think so, largely becauseof the practical
utility that dependablemanipulandahave for our ability to surviveand prosper.
IUTouldwe reinvent the experimentas a method for investigatingsuch causes?
I
AND GENERALTZED
32 | 1. EXPERTMENTS CAUSAL
TNFERENCE
fltHIS BooK covers five central topics across its 13 chapters. The first topic
| (Chapter 1) deals with our general understanding of descriptive causation and
I experimentation. The second (Chapters 2 and 3) deals with the types of valid-
ity and the specific validity threats associatedwith this understanding. The third
(Chapters 4 through 7) deals with quasi-experimentsand illustrates how combin-
ing design features can facilitate better causal inference. The fourth (Chapters 8
through L0) concerns randomized experiments and stressesthe factors that im-
pede and promote their implementation. The fifth (Chapters 11 through L3) deals
with causal generalization, both theoretically and as concerns the conduct of in-
dividual studies and programs of research.The purpose of this last chapter is to
critically assesssome of the assumptions that have gone into these five topics, es-
pecially the assumptions that critics have found obiectionable or that we antici-
'We
pate they will find objectionable. organize the discussionaround each of the
five topics and then briefly justify why we did not deal more extensivelywith non-
experimental methods for assessingcausation.
I7e do not delude ourselvesthat we can be the best explicators of our own as-
sumptions. Our critics can do that task better. But we want to be as comprehen-
sive
srve and
an(l as explicit
explclt as we can.
can. This
I nrs is rn part
ls in part because
becausewe
we are
are convinced
convrnced of
ot the
the ad-
acl-
vantages of falsification as a major component of any epistemology for the social
sciences,and forcing out one's assumptions and confronting them is one part of
falsification. But it is also becausewe would like to stimulate critical debateabout
theseassumptionsso that we can learn from those who would challengeour think-
456
AND EXPERIMENTATION
CAUSATION rct
|
ENTATION
AND EXPERIM
CAUSATION
objections to experimentation are probably the most prevalent and virulent. Few
educational researchersseemto object to the following substantiveconclusions of
the form that A dependably causesB: small schools are better than large ones;
time-on-task raises achievement; summer school raises test scores;school deseg-
regation hardly affects achievement but does increaseWhite flight; and assigning
and grading homework raises achievement.The critics also do not seemto object
to other conclusions involving very simple causal contingencies: reducing class
"sizable"
size increasesachievement,but only if the amount of change is and to a
level under 20; or Catholic schools are superior to public ones, but only in the in-
ner city and not in the suburbs and then most noticeably in graduation rates rather
than in achievementtest scores. ,
The primary iustification for such oversimplifications-and for the use of the
experiments that test them-is that some moderators of effects are of minor rele-
vance to policy and theory even if they marginally improve explanation. The most
important contingencies are usually those that modify the sign of a causal rela-
tionship rather than its magnitude. Sign changesimply that a treatment is benefi-
cial in some circumstancesbut might be harmful in others. This is quite different
from identifying circumstancesthat influence just how positive an effect might be.
Policy-makers are often willing to advocate an overall change,even if they suspect
it has different-sizedpositive effects for different groups, as long as the effects are
rarely negative. But if some groups will be positively affected and others nega-
tively political actors are loath to prescribe different treatments for different
groups becauserivalries and jealousies often ensue. Theoreticians also probably
pay more attention to causal relationships that differ in causal sign becausethis
result implies that one can identify the boundary conditions that impel such a dis-
parate data pattern.
Of course, we do not advocate ignoring all causal contingencies.For exam-
ple, physicians routinely prescribe one of severalpossibleinterventions for a given
diagnosis.The exact choice may depend on the diagnosis,test results,patient pref-
erences, insurance resources, and the availability of treatments in the patient's
area. However, the costs of such a contingent system are high. In part to limit the
number of relevant contingencies,physicians specialize,andwithin their own spe-
cialty they undergo extensivetraining to enable them to make thesecontingent de-
cisions. Even then, substantial judgment is still required to cover the many situa-
tions in which causal contingencies are ambiguous or in dispute. In many other
policy domains it would also be costly to implement the financial, management,
and cultural changesthat a truly contingent system would require even if the req-
uisite knowledge were available. Taking such a contingent approach to its logical
extremes would entail in education, for example, that individual tutoring become
dav.Studentsand instructorswould haveto be carefullymatched
the order of the day. matched
for overlap in teachingand learning skills and in the curriculum supportsthey
would need.
tilTithinlimits, some moderators can be studied experimentallSeither by
measuringthe moderator so it can be testedduring analysisor by deliberately
I Ot'
AND EXPERIMENTATION
CAU5ATION
experi-
varying it in the next study in a program of research'In conductingsuch
tak-
ments,onemovesawayfrom thethik-bo" experimentsof yesteryeartoward
more seriouslyand toward routinely study!1g them by,
ing causalcontingencies
foi .""-ple, disaggregating the treatmentto examineits causallyeffectivecom-
ponents,iir"ggt.glting the effect,toexamineits causallyimpactedcomponents,
variables,and
.ondrr.ting ,n"ty*r ofi.-ographic and psychologicalmoderator
affects
exploringlhe causalpathwa-ysihtooghwhjch (parts.of) the treatment
in a singleexperimentis not possi-
lparts of) the outcomJ.To do all of this well
tl.. brrtto do someof it well is possibleand desirable.
of E4periments
Criticisms
Epistemological
we have
In highlightingstatisticalconclusionvalidity and in-selectingexamples,
testing'
often linked causaldescriptionto quantitativemethodsand hypothesis
positivism'
Many criticswill (wrongly)r.. this asimplying a discreditedtheory of
positivismre-
As a philosophyof scieniefirst outlined in the early L9th century'
about unobservables,and equated
1.ct.d' metaphysicalspeculations,especially school of
lrro*t.ag. *lih descriptionsof e*periencedphenomena-A narrower
realism
logical pisitivism .*.rg.d in the eatly 20th century that also rejected
logic form
*til. "lro .-phasizing Ih. ,rr. of data-theoryconnectionsin predicate
Both thesere-
""J " fr.f.r.r.. for p"redictingphenomenaover explainingthem'
lated epistemologies *.r. lonf ago discredited,especiallyas explanationsof how
this basis'How-
scienceop.r"trr.*so few criticsseriouslycritici'e experimentson
to attack
ever,many critics use the term positiuismwith lesshistorical fidelity
1985)'
quantitativesocialsciencemethodsin genera-l(e'g', Lincoln & Guba,
use of quantification
liuilding on the rejectionof logicalpositivism,they reiectthe
and forLal logic in observatiron, measurement,and hypothesistesting.Because
of posi-
theselast featuresare part of experiments,to reiectthis looseconception
are nu-
tivism entailsrejectingexperiments.However,the errorsin suchcriticisms
(like the idea that
merous.For example,to ,eject a specificfeatureof positivism
only permissiblelinks betweendata and
f,r"rrtifi.rtion and p redicatelogicare the
tiheory;doesnot nJcessarily imlly reiectingall relatedand more generalproposi-
testing
tions jsuch asthe notion that somekinds of quantificationand hypothesis
outlined more such er-
may be usefulfor knowledgegrowth).Ife and othershave
rors elsewhere (Phillips,1990;Shadish,I995al'
other epistemological criticismsof experimentationcitethe work of historians
and'wool-
of sciencesuchasKuh"n(1,g62),ofsociologistsof sciencesuchasLatour
tend
gar ltiZll "rrd of fhiloroph.ir of scienceiuchas Harr6'(1931).Thesecritics
of theories,the notion that
to focuson threethings.orre.i, the incommensurability
theoriesare neverper"fectly specifiedand so can alwaysbe reinterpreted.As a re-
be reiected'its
sult, when disconfirmingdata seemto imply that a theory should
poriolut., can insteadbI reworkedin order to make the theory and observations
to the
consistentwith eachother.This is usuallydoneby addingnew contingencies
460 | 14.A CRIT|CAL
ASSESSMENT
OF OURASSUMPTTONS
I
theory that limit the conditions under which it is thought to hold. A second cri-
tique is of the assumption that experimental observations can be used as truth
'We
tests. would like observations to be objective assessmentsthat can adjudicate
between different theoretical explanations of a phenomenon. But in practice, ob-
servationsare not theory neutral; they are open to multiple interpretations that in-
clude such irrelevanciesas the researcher'shopes, dreams, and predilections. The
consequenceis that observations rarely result in definitive hypothesistests.The fi-
nal criticism follows from the many behavioral and cognitive inconsistenciesbe-
tween what scientists do in practice and what scientific norms prescribe they
should do. Descriptions of scientists' behavior in laboratories reveal them as
choosing to do particular experiments becausethey have an intuition about a re-
lationship, or they are simply curious to seewhat happens, or they want to play
with a new piece of equipment they happen to find lying around. Their impetus,
therefore, is not a hypothesis carefully deduced from a theory that they then test
by means of careful observation.
Although these critiques have some credibilitg they are overgeneralized.Few
experimentersbelievethat their work yields definitive results even after it has been
subjected to professional review. Further, though these philosophical, historical,
and social critiques complicate what a "fact" means for any scientific method,
nonethelessmany relationships have stubbornly recurred despite changesassoci-
ated with the substantive theories, methods, and researcherbiasesthat first gen-
erated them. Observations may never achieve the status of "facts," but many of
them are so stubbornly replicable that they may be consideredas though they were
facts. For experimenters, the trick is to make sure that observations are not im-
pregnated with just one theory, and this is done by building multiple theories into
observationsand by valuing independent replications, especiallythose of sub-
stantive critics-what we have elsewherecalled critical multiplism (Cook, 1985;
Shadish,'1.989, 1994).
Although causal claims can never be definitively tested and proven, individ-
ual experiments still manage to probe such claims. For example, if a study pro-
duces negative results, it is often the casethat program developersand other ad-
vocates then bring up methodological and substantive contingenciesthat might
have changedthe result. For instance, they might contend that a different outcome
measure or population would have led to a different conclusion. Subsequentstud-
ies then probe these alternatives and, if they again prove negative, lead to yet an-
other round of probes of whatever new explanatory possibilities have emerged.
After a time, this process runs out of steam, so particularistic are the contingen-
cies that remain to be examined. It is as though a consensusemerges:"The causal
relationship was not obtained under many conditions. The conditions that remain
to be examined are so circumscribed that the intervention will not be worth much
'W'e
even if it is effectiveunder these conditions. " agreethat this processis as much
or more social than logical. But the reality of elastic theory does not mean that de-
cisions about causal hypotheses are only social and devoid of all empirical and
I
logical content.
I
I
t
I
J
| +er
AND EXPERIMENTATION
CAUSATION
NeglectedAncillarYQuestions
Our focus on causalquestionswithin an experimentalframework neglectsmany
other questionsthat arerelevantto causation.Theseincludequestionsabout how
to decideon the importanceor leverageof any singlecausalquestion.This could
entail exploringwhethera causalquestionis evenwarranted,as it often is not at
the early sa"g.-ofdevelopmentof an issue.Or it could entail exploringwhat type
of c".rsalquestionis moie important-one that fills an identifiedhole in somelit-
erature,o, orr. that setsout to identify specificboundary conditionslimiting a
causalconnection,or one that probesthe validity of a centralassumptionheld by
all the theoristsand researchers within a field, or one that reducesuncertainty
about an important decisionwhen formerly uncertaintywas high. Our approach
alsoneglectsthe realitythat how oneformulatesa descriptivecausalquestionusu-
aily enLils meetingsomestakeholders'interestsin the socialresearchmore than
those of others.TLus to ask about the effectsof a national program meetsthe
needsof Congressional staffs,the media,and policy wonks to learnaboutwhether
the program"*orks. But it can fail to meet the needsof local practitionerswho
,rro"lly"*"nt to know about the effectiveness of microelementswithin the pro-
gram ,o thut they can usethis knowledgeto improve their daily practice.-Inmore
Ih.or.ti."l work, to ask how some interventionaffectspersonalself-efficacyis
likely to promote individuals'autonomyneeds,whereasto ask about the effects
of a'persoasivecommunicationdesignedto changeattitudescould well cater to
t
462 14.A CR|T|CAL
ASSESSMENT
OFOURASSUMPT|ONS
|
the needs of those who would limit or manipulate such autonomy. Our narrow
technical approach to causation also neglectedissuesrelated to how such causal
knowledge might be used and misused. It gave short shrift to a systematic analy-
sis of the kinds of causal questions that can and cannot be answered through ex-
periments. \7hat about the effects of abortion, divorce, stable cohabitation, birth
out of wedlock, and other possibly harmful events that we cannot ethically ma-
nipulate? What about the effects of class,race, and gender that are not amenable
'What
to experimentation? about the effects of historical occurrencesthat can be
studied only by using time-seriesmethods on whatever variables might or might
not be in the archives?Of what use, one might ask, is a method that cannot get at
some of the most important phenomena that shape our social world, often over
generations,as in the caseof race, class,and gender?
Many statisticians now consider questions about things that cannot be ma-
nipulated as being beyond causal analysis,so closely do they link manipulation to
causation. To them, the cause must be at least potentially manipulable, even if it
is not actually manipulated in a given observational study. Thus they would not
consider race ^ cause, though they would speak of the causal analysis of race in
studies in which Black and White couples are, say, randomly assignedto visiting
rental units in order to seeif the refusal rates vary, or that entail chemically chang-
ing skin color to seehow individuals are responded to differently as a function of
pigmentation, or that systematicallyvaried the racial mix of studentsin schools or
classrooms in order to study teacher responsesand student performance. Many
critics do not like so tight a coupling of manipulation and causation. For exam-
ple, those who do status attainment researchconsider it obvious that race causally
influences how teachers treat individual minority students and thus affects how
well these children do in school and therefore what jobs they get and what
prospects their own children will subsequentlyhave. So this coupling of causeto
manipulation is a real limit of an experimental approach to causation. Although
we like the coupling of causation and manipulation for purposes of defining ex-
periments, we do not seeit as necessaryto all useful forms of cause.
VALIDITY
Objectionsto InternalValidity
There are severalcriticismsof Campbell's(1957) validity typology and its exten-
sions(Gadenne,1976;Kruglanski& Kroy, 1.976;Hultsch 6cHickey,1978;Cron-
'We
bach, 1982; Cronbachet al., 1980). start first with two criticismsof internal
validity raisedby Cronbach(1982)and to a lbsserextentby Kruglanskiand Kroy
(1'976):(1) an atheoreticallydefinedinternal validity (A causesB) is trivial with-
out referenceto constructs;and (2) causationin singleinstancesis impossible,in-
cludingin singleexperiments.
vALtDtrY nol
I
Objections
to Causation
in SingleExperiments
A second criticism of internal validity deniesthe possibility of inferring causation
in a single experiment. Cronbach (1982) says that the important feature of cau-
sation is the "progressivelocalizationof a cause" (Mackie, 1974, p.73) over mul-
J
vALrDrry otu
|
tiple experimentsin a program of researchin which the uncertainties about the es-
sential i."t.rr.r of the cause are reduced to the point at which one can character-
ize exacflywhat the causeis and is not. Indeed, much philosophy of causation as-
serts that we only recognize causes through observing multiple instances of a
putative causal relationship, although philosophers differ as to whether the mech-
anism for recognition involves logical laws or empirical regularities (Beauchamp,
1974;P. White, 1990).
However, some philosophers do defend the position that causescan be in-
ferred in singleinstances(e.g.,Davidson, 1,967;Ducasse'1,95L1' Madden & Hum-
ber, L97'1,).A good example is causation in the law (e.g.,Hart & Honore, 1985)'
by which we judge whether or not one person, say, caused the death of another
despitethe fact that the defendant may 4ever before have been on trial for a crime.
The verdict requires a plausible casethat (among other things) the defendantb ac-
tions precededlhe death of the victim, that those actions were related to the death,
that other potential causesof the death are implausible, and that the death would
not have occurred had the defendant not taken those actions-the very logic of
causal relationships and counterfactualsthat we outlined in Chapter 1. In fact, the
defendant'scriminal history will often be specifically excluded from consideration
in iudging guilt during the trial. The lessonis clear. Although we may learn more
"bo,rt ."nsation from multiple than from single experiments, we can rnf.ercause
in single experiments.Indeed, experimenterswill do so whether we tell them to or
not. Providing them with conceptual help in doing so is a virtue, not a vice; fail-
ing to do so is a major flaw in a theory of cause-probing methods.
Of course, individual experiments virtually always use prior concepts from
other experiments.However, such prior conceptualizations are entirely consistent
with the claim that internal validity is about causal claims in single experiments.
If it were not (at least partly) about single experiments, there would be no point
to doing the experiment, for the prior conceptualization would successfullypre-
dict what will be observed.The possibility that the data will not support the prior
conceptualization makes internal validity essential.Further, prior conceptualiza-
tions are not logically necessary;we can experiment to discover effects that we
"The physicist George Darwin used
have no prior conceptual structure to expect:
to say tliat once in a while one should do a completely crazy experiment, like
blowing the trumper to the tulips every morning for a month. Probably nothing
wiil hafpen, but if something did happen, that would be a stupendousdiscovery"
(Hacking, L983, p. 15a). But we would still need internal validity to guide us in
judging if the trumpets had an effect.
ObjectionsConcerning Between
the Discrimination
ConstructValidityand ExternalValidity
Although we traced the history of the present validity system briefly in Chapter 2,
readers may want additional historical perspectiveon why we made the changes
we made in the present book regarding construct and external validity. Both
Campbell (1957) and Campbell and Stanley(1963) only usedthe phraseexternal
validitS which they defined as inferring to what populations, settings,treatment
variables, and measurement variables an effect can be generalized.They did not
rcfer at all to construct validity. However, from his subsequentwritings (Camp-
bell, 1986), it is clear Campbell thought of construct validity as being part of ex-
ternal validity. In Campbell and Stanley therefore, external validity subsumed
generalizing from researchoperations about persons, settings,causes,and effects
for the purposes of labeling theseparticulars in more abstract terms, and also gen-
eralizing by identifying sourcesof variation in causal relationships that are attrib-
utable to person, setting, cause, and effect factors. All subsequentconceptualiza-
tions also share the same generic strategy based on sampling instancesof persons,
settings, causes,and effects and then evaluating them for their presumed corre-
spondenceto targets of inference.
In Campbell and Stanley'sformulation, person, setting, cause,and effect cat-
egories share two basic similarities despite their surface differences-to wit, all of
them have both ostensive qualities and construct representations.Populations of
persons or settings are composed of units that are obviously individually osten-
sive. This capacity to point to individual persons and settings, especially when
they are known to belong in a referent category permits them to be readily enu-
merated and selectedfor study in the formal ways that sampling statisticianspre-
fer. By contrast, although individual measures (e.g., the Beck Depression Inven-
tory) and treatments (e.g., a syringe full of a vaccine) are also ostensive,efforts to
enumerate all existing ways of measuring or manipulating such measuresand
treatments are much more rare (e.g.,Bloom, L956; Ciarlo et al., 1986; Steiner&
Gingrich, 2000). The reason is that researchersprefer to use substantivetheory to
determine which attributes a treatment or outcome measureshould contain in any
.J
vALrDtrYI oe,
given studS recognizing that scholars often disagreeabout the relevant attributes
of th. higher order entity and of the supposed best operations to representthem.
None of ihis negatesthe reality that populations of persons or settingsare also de-
fined in part by the theoretical constructs used to refer to them, just like treatments
and outiomes; they also have multiple attributes that can be legitimately con-
'!(hat,
tested. for instance, is the American population? \7hile a legal definition
surely exists,it is not inviolate. The German conception of nationality allows that
the gieat grandchildren of a German are Germans even if their parents and grand-
p"r*t, have not claimed German nationality. This is not possible for Americans.
And why privilege alegaldefinition? A cultural conception might admit as Amer-
ican all thor. illegal immigrants who have been in the United Statesfor decades
and it might e*cl.rde those American adults with passports who have never lived
in the United States. Given that person's,settings, treatments, and outcomes all
have both construct and ostensive qualities, it is no surprise that Campbell and
Stanley did not distinguish between construct and external validity.
Cook and Camptell, however, did distinguish between the two. Their un-
stated rationale for the distinction was mostly pragmatic-to facilitate memory
for the very long list of threats that, with the additions they made' would have
had to fit under bampbell and Stanley'sumbrella conception of external validity.
In their theoretical diicussion, Cook and Campbell associatedconstruct validity
with generalizingto causesand effects, and external validity with generalizing to
and across persons, settings, and times. Their choice of terms explicitly refer-
encedCronbach and Meehl (1955) who used construct and construct validity in
"about higher-order constructs from re-
measurementtheory to justify inferences
search operations'; lcook & Campbel| 1,979, p. 3S). Likewise, Cook and
Campbeli associatedthe terms population and external ualidity with sampling
theory and the formal and purposive ways in which researchersselect instances
of persons and settings. But to complicate matters, Cook and Campbell also
"all aspectsof the researchrequire naming samples in
brlefly acknowledged that
gener-alizable termi, including samplesof peoples and settings as well as samples
of -r"r,rres or manipulations" (p. 59). And in listing their external validity
threats as statistical inieractions between a treatment and population, they linked
external validity more to generalizing across populations than to generalizing to
them. Also, their construct validity threats were listed in ways that emphasized
generalizing to cause and effect constructs. Generalizing across different causes
ind effect, *", listed as external validity becausethis task does not involve at-
tributing meaning to a particular measure or manipulation. To read the threats
in Cook and Campbell, external validity is about generalizing acrosspopulations
of persons and settings and across different cause and effect constructs, while
construct validity is about generalizing to causesand effects.Where, then, is gen-
era\zing from samples of persons or settings to their referent populations? The
text disiussesthis as a matter of external validitg but this classification is not ap-
parent in the list of validity threats. A system is neededthat can improve on Cook
and Campbell's partial confounding between objects of generalization (causes
468 ASSESSMENT
14.A CRITICAL OF OURASSUMPTIONS
and effects versus persons and settings) and functions of generalization (general-
izing to higher-order constructs from researchoperations versus inferring the de-
greeof replicationacrossdifferent constructsand populations).
This book usessucha functional approachto differentiateconstructvalidity
from externalvalidity. It equatesconstructvalidity with labelingresearchopera-
tions, and externalvalidity with sourcesof variation in causalrelationships.This
new formulation subsumesall of the old. Thus, Cook and Campbellt under- ,i-{11
standingof constructvalidity asgeneralizingfrom manipulationsand measures to f i..
t
.,.J
I oo,
vALrDrrY
tributes may be pertinent labels for some of the inferencesbeing made. Hence, we
usually have to rely on the assumption that, becausegender samplescome from
the same physical setting, they are comparable on all background characteristics
that might be correlated with the outcome. Becausethis assumption cannot be
fully testedand is ^nyw^y often false-as in the hypothetical example above-this
means rhat we could and should measure all the potential confounds within the
limits of our theoretical knowledge to suggestthem, and that we should also use
these measuresin the analysis to reduce confounding.
Even with acknowledged confounding, sample-specific differences in effect
sizesmay still allow us to conclude that a causal relationship varies by something
associatedwith gender.This is a useful conclusion for preventing premature over-
generalization.Iilith more breakdownq, confounded or not, one can even get a
senseof the percentageof contrastsacrosswhich a causal relationship does and
does not hold. But without further work, the populations across which the rela-
tionship varies are incompletely identified. The value of identifying them better is
particularly salient when some effect sizescannot be distinguished from zero. Al-
though this clearly identifies a nonuniversal causal relationship, it does not ad-
vance theory or practice by specifying the labeled boundary conditions over which
a causal relationship fails to hold. Knowledge gains are also modest from gener-
alization strategiesthat do not explicitly contrast effect sizes.Thus, when differ-
ent populations are lumped together in a single hypothesis test, researcherscan
learn how large a causal relationship is despite the many unexamined sources of
variation built into the analysis. But they cannot accurately identify which con-
structs do and do not co-determine the relationship's size. Construct validity adds
useful specificity to external validity concerns, but it is not a necessarycondition
for external validity.'We can generalize across entities known to be confounded'
albeit lessusefully than acrossaccurately labeled entities.
This last point is similar to the one raised earlier to counter the assertion of
Gadenne (L9761and Kruglanski and Kroy (1976) that internal validity requires
the high consrruct validity of both causeand effect. They assertthat all scienceis
"something causedsome-
about constructs, and so it has no value to conclude that
thing sfss"-1hs result that would follow if we did a technically exemplary ran-
domized experiment with correspondingly high internal validity but the causeand
effect were not labeled. Nonetheless, a causal relationship is demonstrably en-
"something reliably causedsomething else" might lead
tailed, and the finding that
to further researchto refine whatever clues are available about the cause and ef-
fect constructs. A similar argument holds for the relationship of construct to ex-
ternal validity. Labels with high construct validity are not necessaryfor internal
or for external validity, but they are useful for both.
Researchersnecessarilyuse the language of constructs (including human and
setting population ones) to frame their research questions and selecttheir repre-
sentationsof constructsin the samplesand measureschosen.If they have designed
their work well and have had some luck, the constructs they begin and end with
will be the same,though critics can challengeany claims they make. However, the
470 14.A CRITICAL OFOURASSUMPTIONS
ASSESSMENT
samplesand constructs might not match we[], and then the task is to examine the
samples and ascertain what they might alternatively stand for. As critics like
Gadenne,Kruglanski,and Kroy havepointedout, suchrelianceon the operational
levelseemsto legitimizeoperationsashavinga life independentof constructs.This
is not the case,though,for operationsare intimatelydependenton interpretations
at all stagesof research.Still, every operation fits some interpretations, however
tentative that referent may be due to poor researchplanning or to nature turning
out to be more complex than the researcher'sinitial theory.
of Personsor settings
should Generalizingfrom a single sample
Be Classifiedas External or Construct Validity?
If a study hasa singlesampleof pers.ons or settings,this samplemust representa
is an issue'Given that construct
population.How ,"nlrrr-pre should be labeled
an issueof constructvalidity?Af-
validity is about rJ.iirrg, i, Itbeling the lample
sincewith a singlesampleit is not
ter all, externalvalidity hardly seemsrelevant
in causalrelationshipswould
immediatelyobvious*n", comparisonof variation
of personsor settingsis treatedas a
be involved.So if g.".t"iit-g fio* a sample
from treatment and out-
matter of constructvalidity analogousto generalizing
highlightsa potential conflict in
come operations,i*o probl.-, "r-ir.. Firstl this
someparts of which saythat gen-
usagein the generalsocialsciencecommunity' va-
eralizationsfrom;;;i; of peopleto its pofulation are a matter of external
lidity, evenwhen ;rh.;;;", ,"y ih", labefingpeopleis a matter of constructva-
in Cook and Campbellthat
lidity. Second,trrir-J".r not fit'with the discussion
personsand settingsas an external
treatsgeneralizingrr.t" irrdiuidrr"lsamplesof
threatsdoesnot explicitly deal
validity matter,thoughtheir list of .*t.*"1 validity
with this and only mentionsinteracti,ons betweenthe treatmentand attributesof
the settingand Person.
selectedfrom.the pop-
The issueis most acutewhen the samplewas randomly
so keento promoterandom sam-
ulation. considerwhy samplingstatisticiansare
Suchsamplingensuresthat the
pling for represe";i"; " *.il-dJrignated universe.
on all measuredand unmeasured
sampleand populatiJndistributionsare identical
Notice that this includesthe popula-
variableswithin the limits of samplingerror. also
randomsamplingguarantees
tion label(whethermoreor less"ccorit.;, which a well
appliesto the ,";;[. K.y tg tle or.i rl*r, of random samplingis having
in samplingtheory and
boundedpop.rl"tiJ., from which to sample,a-requirement
many well boundedpopulations
somethingoften obviousin practice.Given that
guarantees that a valid populationla-
are alsowell tabeied,r""a.- samplingthen
For instance'the population of
bel can equallyvalidly be applied,o itt. saripl..
known and is obviouslycorrectly
telephoneprefixesor.d i' tlie city of Chicagols
digit dialing frol that list of
labeled.Hence,i *""fa be difficuli. ,rrJt"ndom telephone
sampleas representing
Chicagopr.fi*., "nJ itt." mislabelthe resulting a clearly
sJction-of Chicago- Given
ownersin Detroii o, orty in the Edgewater
the samplelabel is the populationla-
boundedpopulationand random saripling,
that no methodis superiorto ran-
bel, which is why samplingstatisticiansbelieve
populationlabelis known'
dom selectio'f- iun.ii"g"tumpleswhen the
OFOURASSUMPTIONS
ASSESSMENT
472 I T+.N CRITICAL
ulation who were not randomly sampledfall into the latter category.Nothing
about externalvalidity,eitherin the presentbook or in its predecessors, requires
that all possibleuariuiion, of externalvalidity interestactuallybe observedin the
study-indeed, it would beimpossibleto do so,and we providedseveralarguments
in Cirapter2 aboutwhy it would not be wise to limit external validity questions
only to variationsactuallyobservedin a study.Of course,in most casesexternal
ualidiry generalizations to things that were not studied are difficult, having to rely
on the .L.r..pt, and methodswe outlined in our grounded theory of generalized
causalinferencein Chapters11 through 13. But it is the great beautyof random
samplingthat it guaran;es that this generalizationwill hold over both sampledand
,rnr"-pl".d p.rr6nr. So it is indeedan externalvalidity questionwhah-e1acausal
relationshipthat hasbeenobservedin a singlerandomsamplewould hold for those
units that were in the populationbut not'in the random sample.
Inthe end,this book treatsthe labelingof a singlesampleof personsor set-
tings asa matterof constructvalidiry whetheror not random samplingis used.It
alsi treatsthe generalizationof causalrelationshipsfrom a singlesampleto un-
observedinstancesasa matterof externalvalidity-againrwhether or not random
samplingwas used.The fact that random sampling(which is associatedwith ex-
,.rrr"l uiiairy in this book) sometimeshappensto facilitatethe constructlabeling
of a sampleis incidentalto the fact that the population label is alreadyknown.
Though many populationlabelsare indeedwell-known, many more are still mat-
,.r, of debate,as reflectedin the exampleswe gavein Chapter3 of whetherper-
sonsshouldbe labeledschizophrenicor settingslabeledas hostilework environ-
ments.In theselatter cases,random samplingmakesno contribution to resolving
debatesabout the applicabilityof thoselabels.Instead,the principlesand meth-
ods we outlinedin Ci"pt.rs 11 through 13 will haveto be brought to bear.And
when random samplinghasnot beenused,thoseprinciplesand methodswill also
haveto be broughito b.". on the externalvalidity problemof generalizingcausal
relationshipsfrom singlesamplesto unobservedinstances.
of the Typology
ObjectionsAbout the Completeness
The first objectionof this kind is that our lists of particularthreatsto validity are
incomplete.Bracht and Glass(1,968),for example,ad-dednew externalvalidity
threatsthat they thought were overlookedby Campbelland Stanley(1,96311' and
more recentlyAiken ind West (1991) pointed to new reactivity threats._ These
challenges "r. i*portant becausethe key to the most confidentcausalconclusions
in our ,f,.ory of validity is the ability to construct a persuasiveargumentthat every
plausibleand identifiedthreat to validity has beenidentifiedand ruled out. How-
iver, thereis no guaranteethat all relevantthreatsto validity havebeenidentified.
Our lists are not divinely ordained,as can be observedfrom the changesin the
threats from Campbel IUST) to Campbell and Stanley (1'963)to Cook and
14.A CRITICAL OF OURASSUMPTIONS
ASSESSMENT
1. We are acutelyaware of, and modestlydismayedat, the many differentusagesof thesevalidity labelsthat have
developedover the years and of the risk that posesfor terminological confusion---eventhough we are responsible
for rnany of thesevariations ourselves.After all, the understandingsof validiry in this book differ from those in
Campbelland Stanley(1963),whoseonly distinctionwas betweeninternal and externalvalidity. They alsodiffer
from Cook and Campbell (7979), in which externalvalidity was concernedwith generalizingto and across
populations of personsand settings,whereasall issuesof generalizingfrom the causeand effect operations
constitutedthe domain of constructvalidity. Further,Campbell(1985) himselfrelabeledinternalvalidiry and
external validiry as local molar causalvalidity and the principle of proximal similarity, respectively.Steppingoutside
Campbell'stradition, Cronbach(1982) usedtheselabelswith yet other meanings.He said internalvalidity is the
problem of generalizingfrom samplesto the domain about which the questionis asked,which soundsmuch like our
construct validity except that he specifically denied any distinction betweenconstruct validiry and external validiry,
using the latter term to refer to generalizingresults to unstudied populations, an issueof extrapolation beyond the
data at hand. Our understandingof external validity includessuch extrapolations as one case,but it is not limited to
that becauseit also has to do with empirically identifying sourcesof variation in an effect sizewhen existing data
allow doing so. Finally, many other authors have casually used all theselabels in completelydifferent ways (Goetz
& LeCompte,1984; Kleinbaum,Kupper, & Morgenstern,1982;Menard, 1991).So in view of all thesevariations,
we urge that theselabels be used only with descriptionsthat make their intended understandingsclear.
::j
!t
t
VALIDTTY 47s
|
i
,l
,I
t
VALIDITY I 477
notes that his intention is not to open the door to the social policing of truth (i.e.,
a test is valid if its social consequencesare good), but ambiguity on this issuehas
nonethelessopened this very door. For example, Kirkhart (1,995)cites Messick as
justification for judging the validity of evaluations by their social consequences:
"Consequential
validity refers here to the soundnessof changeexerted on systems
by evaluationand the extent to which thosechangesare just" (p.a).This notion is
risky becausethe most powerful arbiter of the soundnessand iustice of social con-
sequencesis the sociopolitical systemin which we live. Depending on the forces in
power in that system at any given time, we may find that what counts as valid is
effectively determined by the political preferencesof those with power.
:
I
:l{
VALIDITYI O''
I
:iil
14.A CRITICAL
ASSESSMENT
OF OURASSUMPTIONS
j
t
I
I
'iil
VALIDITYI +ET
3. The fact that different people might have different beliefs about the same referent is sometimes cited as violating
this maxim, but it need not do so. For example, if the knowledge claim being validated is "John views the program
as effective but Mary views it as ineffective," the claim can be true even though the views of John and Mary are
contradictory.
ii
j
VALIDITY I 483
Q UASI.EXPERIM ENTATION
Criteriafor RulingOut Threats:
The Centralityof FuzzyPlausibility
In a randomized experiment in which all groups are treated in the sameway excepr
for treatment assignment,very few assumptionsneed to be made about ro,rr.", of
bias. And those that are made are clear and can be easily tested,particularly as con-
cerns the fidelity of the original assignment process and its subsequentmainte-
nance. Not surprisinglS statisticiansprefer methods in which the assumptionsare
few, transparent, and testable. Quasi-experiments, however, rely heavily on re-
searcheriudgments about assumptions, especiallyon the fuzzy but indispensable
concept of plausibility. Judgments about plausibility are neededfor deciding which
of the many threats to validity are relevant in a given study for deciding whether
a particular designelement is capable of ruling out a given threat, for estimating by
how much the bias might have been reduced, and for assessingwhether multiple
threats that might have been only partially adjusted for might add up to a total bias
greater than the effect size the researcher is inclined to claim. Vith quasi-
experiments, the relevant assumptions are numerous, their plausibility is less evi-
dent, and their single and joint effectsare lesseasily modeled. We acknowledgethe
fuzzy way in which particular internal validity threats are often ruled out, and it is
becauseof this that we too prefer randomized experiments (and regressiondiscon-
tinuity designs)over most of their quasi-experimentalalternatives.
But quasi-experiments vary among themselveswith respect to the number,
transparencg and testability of assumptions. Indeed, we deliberately ordered the
chapters on quasi-experiments to reflect the increase in inferential power that
comes from moving from designs without a pretest or without a comparison
group to those with both, to those based on an interrupted time series,and from
there to regression discontinuity and random assignment.Within most of these
chapters we also illustrated how inferencescan be improved by adding design el-
ements-more pretest observation points, better stable matching, replication and
systematic removal of the treatment, multiple control groups, and nonequivalent
dependentvariables. In a sense,the plan of the four chapters on quasi-experiments
reflects two purposes. One is to show how the number, transparency and testa-
bility of assumptions varies by type of quasi-experimental design so that, in the
best of quasi-experiments,internal validity is not much worse than with the ran-
domized experiment. The other is to get students of quasi-experimentsto be more
sparing with the use of this overly general label, for it threatens to tar all quasi-
t
QUASI-EXPERIMENTATION+SS
|
to the
experimentswith the samenegativebrush. As scholarswho have contributed
institution alization of the t i^ quoti-experiment, we feel a lot of ambivalence
the random-
about our role. Scholarsneed to itrint critically about alternatives to
la-
ized experiment, and from this need arisesthe need for the quasi-experimental
under the
bel. But all instancesof quasi-experimentaldesignshould not be brought
the best studies do
sameunduly broad quasi-experimentalumbrella if attributes of
not closely match the weaker attributes of the field writ large.
use of
Statisticians seek to make their assumptions transparent through the
have resisted this strat-
formal models laid out as formulae. For the most part, we
very con-
egy becauseit backfires with so many readers,alienating them from the
words in-
.!pt.r"t issuesthe formul ae aredesignedto make evident.'We have used
cognoscenti'
stead.There is a cost to this, and not jupt in the distaste of statistical
The
particularly those whose own research has emphasized statistical models-
to formally
main cost is that our narrative approach makes it more difficult
the alternative
demonstrate how much fewer and more evident and more testable
quasi-
interpretations became as we moved from the weaker to the stronger
acrossthe
.*p.ri-.rrts, both within the relevant quasi-experimental chapters and
'We
set of them. regret this, but do not apologize for the accessibility we tried to
Fortu-
create by minimirirrg the use of Greek symbols and Roman subscripts.
to develop
nately, this deficit is not absolute, as both we and others have worked
in partic-
meth;ds that can be used to measurethe size of particular threats' both
1998; Shadish, 2000) and
ular studies(e.g.,Gastwirth et al., L994;Shadishet al.,
Posavac,6c
in sets of studiis (e.g.,Kazdin 6c Bass, 1989; Miller, Turner, Tindale,
& Putnam,t982\.
& Rubin,1,978;Willson
Dugoni,1,991;Ror."nitt.t our
Further,
statistical
narrative approach has a significant advantage over a more narrowly
threats
emphasisii allows us to addressa broad er array of qualitatively different
that there-
to validitS threats for which no statistical measure is yet available and
quantification.
fore mighi otherwise be overlooked with too strict an emphasison
at all
Better to h"u. imprecise attention to plausibility than to have no attention
paid to many imptrtant threats just becausethey cannot be well measured'
PatternMatchingas a ProblematicCriterion
This book is more explicitthan its predecessors about the desirabilityof imbuing
a causalhypothesiswith multiple tistable implicationsin the data, providedthat
we
they servett reducethe viability of alternativecausalexplanations.In a sense'
havesoughtto substitutea pattern-matchingme{rod-ology'We for the u-sualassessment
of wheth-era few means,oft.n only fwo, reliably differ. do this not because
num-
.o-pl.*ity itself is a desideratumin science.To the contrary,simpliciry in the
be, of questionsaskedand methodsusedis highly prizedin science. The simplicity
well.
of ,arrjomized experimentsfor descriptivecausalinferenceillustratesthis
However,the samesimple circumstancedoes not hold with quasi-experiments.
With them. we haveassirtedthat causalinferenceis improvedthe more specific,
488 | ro.o cRtlcALAssEssMENT
oF ouRAssuMploNs
RANDOMIZED
EXPERIMENTS
This sectionlistsobjectionsthat havebeenraisedto doingrandomizedexperiments,
and our analysisof the more and lesslegitimateissuesthat theseobiectionsraise.
Experiments
CannotBe Successfully
lmplemented
Even a little exposure to large-scalesocial experimentation shows that treatments
are often improperly or incompletely implemented and that differential attrition
often occurs. Organizational obstaclesto experiments are many. They include the
reality that different actors vary in the priority they attribute to random assign-
ment, that some interventions seem disruptive at all levels of the organization,
and that those at the point of service delivery often find the treatment require-
ments a nuisance addition to their aheady overburdened daily routine. Then
there are sometimes treatment crossovers,as units in the control condition adopt
or adapt components from the treatment or as those in a treatment group are ex-
posed to some but not all of these same components. These criticisms suggestthat
the correct comparison is not between the randomized experiment and better
quasi-experiments when each is implemented perfectly but rather between the
randomized experiment as it is often imperfectly implemented and better quasi-
experiments. Indeed, implementation can sometimes be better in the quasi-
experiment if the decision not to randomize is based on fears of treatment degra-
dation. This argument cannot be addressedwell becauseit dependson specifying
the nature and degree of degradation and the kind of quasi-experimental alter-
native. But taken to its extreme it suggeststhat randomized experiments have no
special warrant in field settings becausethere is no evidencethat they are stronger
than other designs in practice (only in theory).
But the situation is probably not so bleak. Methods for preventing and cop-
ing with treatment degradation are improving rapidly (seeChapter 10, this vol-
I AAS
EXPERIMENTS
RANDOMIZED
random assign-
umel Boru ch,1997;Gueron,1,999;Orr, L999).More important,
with the
-.n, may still createa superiorcounterfactualto its alternativeseven
(1'9961foundthat,
flaws mentionedherein.FLr e*ample,Shadishand Ragsdale
without attrition, randomized experi-
.o-p"..d with randomized."p..i-.tts
nonrandom-
mentswith attrition still yieldedbetter effectsizeestimatesthan did
ran-
ized experiments.Sometimes,of course,an alternativeto severelydegraded
a control'
domizaiion will be best,such as a strong interruptedtime serieswith
poor rule to fol-
But routine rejectionof degradedrandomizedexperimentsis a
to
l,o*; it takescarefulstudy and judgmentto decide.Further,many alternatives
flaws that
experimentationare themselu.i ,ob;..t to treatmentimplementation
thieatenthe validity'weof inferencesfrom them. Attrition and treatmentcrossovers
also occur in them. also suspectthat implementationflaws are salientin ex-
hav6beenaround so long and experimenters
f.ri-errt"tion becauseexperiments the quality
"r. .o critical of eachothlr's work. By contrast,criteria for assessing
(e'g',Datta,
of implementationand resultsfrom othermethodsarefar more recent
lesssubjected
D97j,and they may thereforebe lesswell developedconceptuallS
to peercriticism,and lessimprovedby the lessonsof experience.
ExperimentationNeedsStrongTheoryand Standardized
TreatmentlmPlementation
rs
Many critics claim that experimentationis more fruitful when an intervention
detailsis
basedon strongsubstantivetheory when implementationof treatment
when im-
faithful to that theor5 when the rlsearchsettingis well managed,and
these
plementationdoes,roi uury much betweenunits' In many field experiments'
organiza'
conditions are not met. For example,schools arclarge, complex, social
iio"r *ith multiple programs,disputatiouspolitics, and conflicting stakeholder
well as
goals.Many progr"*, a"reimplementedvariablyacrossschooldistricts,as
of standard
f.ror, ..hoth, .Lrrroo-r, arri ,t.rdents.Therecan be no presumPli9n
1'977)'
implementationor fidelity to programtheory (Berman& Mclaughlin,
well-
But thesecriticismsur., i' fa-ct,misplaced.Experimentsdo not require
implementa-
specifiedprogram theories,good program management,standard
a contri-
tion, or treatmentsthat are tJtally ?aithful to theory' Experimentsmake.
makesa
bution when they simplyprobewhetheran intervention-as-implemented
preceding
marginal improvem.tttt.yord other backgroundvariability. Still, the
suggests
fa.tJ* can ieducestatisticalpower and so cloud causalinference.This
experiments should:
that in settingsin which *or. of these conditions hold,
(L) uselargesamplesto detecteffects;(2) take painsto reducethe influenceof ex-
ma-
traneousvariation either by designor through measurementand statistical
worth study-
nipulation; and (3) studyimplementationquality both as a variable
implement
i"g * its own right in oid.r to ascertainwhich settingsand providers
treat-
thl interventionbetterand asa mediatorto seehow implementationcarries
ment effectsto outcome.
490 | r+.a cRtTtcAL
ASSESSMENT
OFOURA5SUMPTIONS
to de-
wrongly concludinga treatmentis effective (p <.05) at the risk of failing
that include
tect true treatment;ffects;(2) recommendingintent-to-treatanalyses
treatment; (3) deni-
as part of the treatmentthoseunits that have neverreceived
gr"ting inferencesthat result from exploring unplanned treatment interactions
with characteristics of units, observations,settings,or times;and (4) rigidly pur-
emerge
suing a priori experimentalquestionswhen other interestingquestions
about
duriig " ,t,rdy. Mort laypersonsuse a more liberal risk calculusto decide
poten-
.u,rrul inferencesin their own lives,as when they considertaking up some
ii"ity lifesavingtherapy.Should not sciencedo the same' be lessconservative?
Snoula it notlt least-sometimes make different tradeoffs betweenprotection
againstincorrectinferencesand the failure to detecttrue effects?
critics further obiectthat experimeptsprioritize descriptiveover explanatory
whether
causation.The criticsin qrrestionwould toleratemore uncertaintyabout
processes
the interventionworks in order to learn more about any explanatory
that havethe potentialto generalize acrossunits, settings'observations,and times'
qualita-
Further,,o-. critics pr.f!, to pursuethis explanatory knowledgeusing
than
tive meihodssimilar io thor. of th. historian,journalist, and ethnographer
more opaque
by meansof, sa5 structuralequation modeling that seemsmuch
than the narrativereportsof theseother fields'
critics alsodislikethe priority that experimentsgiveto providing policymak-
real-time
ers with ofren belated"rrri.r, about what works insteadof providing
are rarely interested in
help to serviceprovidersin local settings.Theseproviders
" torrg-a.tayed r,rrnmaryofwhat, ptogt"- has.achieved. They often preferre-
elements
ceiving.o.riin,ro.rsfeedbackabouttheir work and especiallyabout those
A recent letter to
oiprJ.ri.. that they can changewithout undue complication'
theNew York Timescapturedthis preference:
to approach issues
Alan Krueger . . claims to eschew value iudgments and wants
his insistenceon postponing changesin ed-
(about educationalreform) empirically. Yet
is itself a value judgment
ucation policy until studiesby iesearchersapproach certainry
in parts of public edu-
in favor of the status quo. In view of the tragic state of affairs
(Petersen, 1999)
cation, his judgment is a most questionableone.
ques-
we agreewith many of thesecriticisms.Among all possible_research
questionsconstituteonly a subset.And of all possiblecausal meth-
tions,cau-sal
ods,experimentation is not relevantio all typesof questionsand all typesof cir-
in
cumstance.One needonly read the list of options and contingenciesoutlined
Ch"p,.r, 9 and L0 to appreciatehow foolhardy it is to advocate experimenta-
"gold standard"that will invariablyresultin
tion on a routine basisas a causal
trade-
clearly interpretableeffect sizes.However,many of the criticisms about
even over-
offs are basedon artificial dichotomies,correctableproblems,-and
im-
simplifications.Experimentscan and should examinereasonsfor variable
They
pl.-.nt"tion, and they should searchto uncover mediating processes'
for the '05
neednot use stringentalpha rates;only statisticaltradition argues
that
level.Nor needonJ restrict dataanalysesonly to the intent-to-treat'though
'aloJ
lueururo.rdeJoru qf,ntu
e sdeld drrprlerrlpuJetur ql1qlv\ ur surerSord rpuar
;o lurluerelm puorq dpursrrd
-Jns eql ol pue qf,ntu oor drlPllu^
Ieurelur aztseqduraapreqr qf,Jeesar;o sururSord
ur pa8raureaABr{stsaSSnsdrolsrq lpql sassau>lea^\
IertuaraJuragr ol uouuane Bur
-llEf, orp arrrtaqrey .(rq8rlrodsaql uI erurl slr a^eq lsntu ad& drlPler d-rela)
/lpll
-EA Jo pnJlsuof, JaAo drrprlerr
lEuJalxa IeuJalur 1o drerurrd eurlnoJ due -ro; 8ur1er
lou eJEeM'T lardu{J uI Jealr oPELua.&\se 'esJnoJIO 'parseSSnsarreqsrrlr.rr tsed
req.a'\sPeaJxadlrear8 sluaut-radxaeldrllnur Ja o senssrdrrpllel leuJalxa pug lrnrls
-uol qloq sserppeol dlneder aql.slsdleue-Eleruur dlrrap
lsoru aeso^4,se 1ng ,sans
-sI asJl{t qroq Sulssarpp" ur r.lf,EarperFrll e^Er{ slueurradxa .paluerg
'larrr dlfsapou sanssr,{rrprlerr lpnprlrpur
Ipuralxo puB lJnrlsuoJ r{foq sserppE ot r{f,rBrsar
letuaurr.ladxeIo srueJSord;o dlpeder agr qrr^\ pessarduneJEaA\ ,1ser1uocdg
'8ur1uru r{uo^\ tanau aJu
lpql stJoapqJfarrnbar sluaurrradxeter{l tsa88nsol luet
-.rodur ool sr s{Jo.&\rpqra 1no Surpurg .sanqod
IErJospaseq-sseua^rpa}Ja alouord
ol lue1Y\ol{1v\sJJelsrlaql Pue srolelsr8al asoql ro; d1-rrlnrrued 'cnerualqord eJoru
uala dlqeqo-rdaru sJel\supJeell tnoqtr^ saurl aturl Buol-opelep qrns .re8uep
IEar
P sI uollEluaurtradxa arntreruardq8noqlly 'sploq uorlenlrs atues aql lsorule pue
'uerSor4
ruaurdolartaq IooqJS aqr uuSaq rauoJ sauef arurs sread 0t sl lI .sra/\,s
-uB ou 'sloogrs peleJelalf,eue8aq
PUPsluolutradxa ou aleq all\ Pue urle-J drua11
eruts sread SI sl fI 'sfteJJe JIeI{l rnoqe sJa^.r{sup Jeolf,ou e^Erl llrls o.&\puu .pasod
-ord arain sJarlf,no^ .splp .uoryo oor
Iooqtrs erurs srcad 0t A ou sr lI IIE ,.raddeq
SFII 'elgBpuedapun sI lEql uollf,auuoJ lusnpr E tnogp suorsnlf,uoo
leraua8 puorg
Sutmerp >lsIJol sI uolluelJetul uE Jo srlaJJear{l uo sarpnts
leluauuadxa Suorls o4
e^Pq ol 'spuno;8 IEIIuePI^ero lerrSoyuo elqrsneldurrdl-realf,arg drrprlerr
leuralur
ol slBarql ssaFn 'saf,uareJur da4;o dtr.rSalureql Sursrulordtuot lnoqlrd\ passoJl eq
louupf, spunoq aruos 'lurod srql ot rrlaqledtuds d11erauafi are am qfinoqrly .ftget
'qrequo.r3 :og5t ''1" ''3'a)
Ir r{luquorJ spoqlau lutuaurradxo ra8uorls aqr Bur
-zrsuqdruasrue.rSo-rd tuory ueql serpnls leluaunradxeuou pue Iuluaur-radxa_isenb
;o dlerrlua uela ro dlrsoru lslsuof, rer{l qf,reasarJo sure-r8ordtuory peuJuel aq IIri\,\
uoupluroJul aroru
InJasn r'rrrpourr
Et'*:u"';rrx;;H:;r:::ilil?
sluerue^ordur leur8rulu JeuIJ-JaAa 'lsa88ns stxel eruos su
ilHt", ",
to 1uo3aqr pur plSrr se
eq tou paau sluaur.radxg '(salqerrul Surlelpau Jo sarnseau Burppe,.8.a)tuaqr;o
^{et salulleluoslnq 'saJJnosalartnber sarnparo-rdasaql
ilV'{ooq srql ur peurllno
spoqrau eqt Sursn pelpreua8aq plnoqs uortuzrlereua8
lesneolnoqp alqrssodse
uolletuJotul qlntu sB puv 'sasseoordSurlelpau pue sauof,lno pepuelurun Burre
-^oJsrp tE parurp uorllellor etvp a^nelrlenb aq plnor{s pue upJ aleql .saruof,lno
pue 'stuerulearl 's8urpas (suos.rad;o sluerussasse dfrpryel lf,nJlsuof, ar{r puu
;o
salduresJo sseualrleluasardar er.lrJo sasdyeue
lulueurr.radxeuou eg osle plnoqs
'paqsqqnd aq uE3 sluaurrradxs
Pue uBr erarll tuoJJ sllnsoJ urrelul .dlsnorl
-nBf, suolsnlf,uol rraqf 3urqf,nof,
PuE seleJ JoJJa ale8rgo.ld lsure8e Surpren8
hlo11u remod lptrrtsrlels pue droagl elrtuetsqns leql luatxe eqr ol suorlrrJal
-uI 'srsdleur auo oq dlorruryap
IEf,Itsllels aroldxa osle uet sJatuJurrradxg pFor{s
Assumean InvalidModel
Experiments
Utilization
of Research
model of decision
To somecritics, experimentsrecreatea naive rational choice
among (the treat-
making. That is, one first lays out the alternativesto choose
one collectsin-
*.rr,rt] then one decideson criteria of merit (the outcomes);then
and finally
formation on eachcriterion for eachtreatment(the data collection),
empirical
one makes a decisionabout the superior alternative.UnfortunatelS
so simpleas the ra-
work on the useof socialsciencedaia showsthat useis not
tional choicemodelsuggests (c. \ufeiss6c Bucuvalas,1980; c''weiss, 1988)'
contexts'ex-
First, evenwhen."-rir. and effectquestionsare askedin decision
ex-
p.ri-.nt"l resultsare still usedalong with other forms of information-from
consensusof a
isting theories,personaltestimony,extrapolationsfrom surveys'
haverecentlybe-
fieldlchims from expertswith intereststo defend,and ideasthat
politics' person-
.o*. trendy.Decisionsare shapedpartly by ideology,interests,
as much made by a
ality, windows of-opportunity, and ualues;and they are
individualor com-
policy-shapirrg.o-*nrrity (cronbachet al., 1980) as by an
overtime asear-
*i,,... Fuither,manydecisionsarenot so much madeasaccreted
maker with few op-
lier decision,.orrrir"in later ones,leavingthe final decision
are available,new
tions ('Weiss,1980). Indeed,by the time ixperimental results
decisionmakersand issuesmay havereplacedold ones.
verdicts
Second,.*p.rirn.nts often yield contestedrather than unanimous
Disputes arise about
that therefore have uncertain implications for decisions.
resultsare valid'
whether the causalquestionswere correctly framed, whether
whetherrelevantoutcomeswere assessed, and whetherthe resultsentail a specific
voucher
decision.For example,reexaminationsof the Milwaukee educational (H'
occurred
;"rdy offereddifferentconclusionsabout whetherand whereeffects
6cDu, 1.999;'Sritte, 1'998,"1'999,2000)' SimilarlS
Fuller,2000;Greene,Peterson,
differenteffect,ir., *.r. generatedfrom the Tennessee classsizeexperiment(Finn
EcAchilles,1.990;Hanusi'ek,1999;Mosteller, Light, 6c Sachs,1996)'Sometimes,
scholarlydisagreements are at issue,but at other timesthe disputesreflectdeeply
conflictedstakeholderinterests.
likely when
Third, short-terminstrumentaluseof experimentaldata is more
it is easierto
the interventionis a minor variant on existingpractice.For example,
criteriafor
changetextbooksin a classroomor pills givenlo patientsor eligibility
or to open
;;;;"* entry than it is to relocatehospitalsto.underservedlocations
state' Becausethe
day-carecentersfor welfare recipientsthroughout an entire
to dramatically
more feasible.tt""g.t are so ,ood.r, in scope,they are lesslikely
on shor-t-termin-
affecttheproble- ih.y address.So critics note that prioritizing
is unlikelyto solve
strumentalchangetendsto preservemost of the statusquo and
that truly twist
tr.rr.hunt social"probl.-s. bf course'thereare someexperiments
from denselypoor
the lion,stail andinvolvebold initiatives.Thus moving families
deviations
inner-citylocationsto the suburbsinvolveda changeof three standard
494 14.A CRIT|CAL
ASSESSMENT
OFOURASSUMPTTONS
|
in the poverty level of the sending and receiving communities, much greater than
what happens when poor families spontaneously move.'S7hethersuch a dramatic
change could ever be used as a model for cleaning out the inner cities of those who
want to move is a moot issue. Many would judge such a policy to be unlikely.
Truly bold experiments have many important rationales; but creating new policies
that look like the treatment soon after the experiment is not one of them.
Fourth, the most frequent use of research may be conceptual rather than in-
strumental, changing how users think about basic assumptions,how they under-
stand contexts, and how they organize'or label ideas. Some conceptual uses are
intentional, as when a person deliberately reads a book on a current problem; for
example, Murray's (1984) book on social policy had such a conceptual impact in
the 1980s, creating a new social policy agenda. But other conceptual usesoccur
in passing, as when a person reads a newspaper story referring to social research.
Such usescan have great long-run impact as new ways of thinking move through
the system, but they rarely change particular short-term decisions.
These arguments against a naive rational decision-making model of experi-
mental usefulnessare compelling. That model is rightly rejected. However, mosr
of the objections are true not just of experiments but of all social sciencemethods.
Consider controversies over the accuracy of the U.S. Census,the entirely descrip-
tive results of which enter into a decision-making process about the apportion-
ment of resourcesthat is complex and highly politically charged. No method of-
fers a direct road to short-term instrumental use. Moreover, the obiections are
exaggerated.In settings such as the U.S. Congress,decision making is sometimes
influenced instrumentally by social scienceinformation (Chelimsky, 1998), and
experiments frequently contribute to that use as part of a researchreview on ef-
fectivenessquestions. Similarlg policy initiatives get recycled, as happened with
school vouchers, so that social science data that were not used in past years are
used later when they become instrumentally relevant to a current issue (Polsby,
1'984; Quirk, 1986).In addition, data about effectivenessinfluence many stake-
holders' thinking even when they do not use the information quickly or instru-
mentally. Indeed, researchsuggeststhat high-quality experiments can confer exrra
'Weiss
credibility among policymakers and decision makers (C. & Bucuvalas,
1980)' as happened with the Tennesseeclasssize study. We should also not forget
that the conceptual use of experiments occurs when the texts used to train pro-
fessionalsin a given field contain results of past studies about successfulpractice
(Leviton 6c Cook, 1983). And using social sciencedata to produce incremental
change is not always trivial. Small changescan yield benefits of hundreds of mil-
lions of dollars (Fienberg,Singer,& Tanur, 1985). SociologistCarol'Weiss,an ad-
vocate of doing research for enlightenment's sake, says that 3 decadesof experi-
ence and her studies of the use of social sciencedata leave her "impressed with the
utility of evaluation findings in stimulating incremental increasesin knowledge
and in program effectiveness.Over time, cumulative incrementsare not such small
potatoes after all" ('Weiss,1998, p. 31,9).Finallg the usefulnessof experimentscan
be increased by the actions outlined earlier in this chapter that involve comple-
ott
EXPERIMENTS
RANDOMIZED I
Heckman claims this kind of problem with the Job taining PartnershipAct
"calls into question
0TPA) evaluation the validity of the experimentalestimates
as a statementabout theJTPAsystemas a whole" (Heckman,1.992, p. ZZ1,).
In many respects,we agreewith thesecriticisms,thoughit is worth noting sev-
eral responsesto them. First, theyassumealack of generalizabllityfrom experi-
ment to policy but that is an empirical question.Somedata suggesrthar general-
ization may be high despite differencesbetweenlab and field (C. Anderson,
LindsaS & Bushman, 1999) or betweenresearchand practice (Shadishet al.,
2000). Second,it can help to implement.treatment underconditionsthat aremore
characteristicof practiceif it doesnot unduly compromiseother researchpriori-
ties. A little forethoughtcan improve the surfacesimilarity of units, trearments,
observations,settings,or timesto their intendedtargets.Third, someof thesecrit-
icismsare true of any researchmethodologyconductedin a limited context,such
as locally conductedcasestudiesor quasi-experiments, becauselocal implemen-
tation issuesalwaysdiffer from large-scaleissues.Fourth, the potentiallydisrup-
tive natureof experimentallymanipulatedinterventionsis sharedby many locally
'rrr"or"h
invented novel programs, euen uhen they are not studied by any
methodologyat all.Innovation inherentlydisrupts,and substantiveliteraturesare
rife with examplesof innovationsthat encounteredpolicy implementationim-
pediments(Shadish,1984).
However,the essentialproblem remainsthat large-scalepolicy implementa-
tion is a singularevent,the effectsof which cannot be fully known exceptby do-
ing the full implementation.A singleexperiment,or evena smallseriesof ri-ilrt
ones,cannotprovidecompleteanswersabout what will happenif the intervention
is adoptedas policy. However,Heckman'scriticism needsreframing.He fails to
distinguishamongvalidity types(statisticalconclusion,internal,.onrtro.., exter-
nal). Doing so makesit clearthat his claim that suchcriticism"calls into question
the validity of the experimentalestimatesasa sratementabout the JTPA,yrt.rr, ",
a whole" (Heckman,1.992, p.221,)is reallyabout externalvalidityand construcr
validity,not statisticalconclusionor internalvalidity.Exceptin thenarrow econo-
metricstraditionthat he understandably cites(Haavelmo,7944;Marschak ,7953;
Tinbergen,1956),few socialexperimentersever claimedthat experimentscould
describethe "systemas a whole"-even Fisher(1935)acknowledged this trade-
off. Further,the econometricsolutionsthat Heckman suggestscannot avoid the
sametradeoffsbetweeninternal and externalvalidity. For example,surveysand
certain quasi-experiments can avoid someproblemsby observingexistinginter-
ventionsthat have aheadybeenwidely implemented,but the validity of tleir es-
timatesof program effectsare suspectand may themselves changeif the program
were imposedevenmore widely as policy.
Addressingthesecriticismsrequiresmultiple lines of evidence-randomized
experimentsof efficacyand effectiveness, nonrandomizedexperimentsthat ob-
serveexistinginterventions,nonexperimentalsurveysto yield estimatesof repre-
sentativeness, statisticalanalysesthat bracketeffectsunder diverseassumpd;ns,
J
II Ot
EXPERIMENTS
RANDOMIZED
lmposingTreatments Flawed
ls Fundamentally
Compared with Encouragingthe Growthof Local
Solutionsto Problems
Experimentsimposetreatmentson recipients.Yet som,elate 20th-centurythought
,.rjg.rt, that imposedsolutionsmay be inferior to solutionsthat are locally gen-
.rJr".a by thoseiho h"n. the problem. Partly,this view is premisedon research
findings of few effectsfor the Great Societysocialprogramsof the 1960sin the
UniteJ States(Murrag 1.984;Rossi, L987),with the presumptionthat a portion
of the failurewas due to the federallyimposednatureof the programs.Partly,the
view reflectsthe successof late 2Oth-centuryfree market economicsand conser-
vative political ideologiescompared with centrally controlled economiesand
more fi|eral political beliefs.Experimentallyimposedtreatmentsare seenin some
quartersas beinginconsistentwith suchthinking'
IronicallS the first objectionis basedon resultsof experiments-if it is true
that impos.i progr"*s do not work, experimentsprovided the evidence.More-
over,thesetro-.ff..t findingsmay havebeenpartly due to methodologicalfailures
of experimentsas they were implementedat that time. Much progressin solving
practicalexperimentalproblemsoccurredafter,and partly in responseto, those
experiments. If so,it is prematureto assumetheseexperimentsdefinitivelydemon-
stiated no effect,especlalygiven our increasedability to detectsmall effectsto-
day ' (D. Greenberg 6c shroder,1,997;LipseSL992;Lipsey6c'Wilson,!993).
iistinguish betweenpolitical-economiccurrencyand the effects
We must also'We
of interventions. know of no comparisonsof, say,the effectsof locally gener-
atedversusimposedsolutions.Indeed,the methodologicalproblemsin doing such
comparisonsare daunting, especiallyaccuratelycategotizinginterventionsinto
the two categoriesand unlonfounding the categorieswith correlatedmethoddif-
ferences.Bariing an unexpectedsolutionto the seeminglyintractableproblemsof
causalinferencein nonrandomizeddesigns,answeringquestionsabout the effects
of locally generatedsolutionsmay requireexactlythe kind of high-qualityexper-
imentatioi being criticized.Though it is likely that locally generatedsolutions
may indeedhavesignificantadvantages,it also is likely that someof thosesolu-
tions will haveto be experimentallyevaluated.
I
CAUSALGENERALIZATION:
AN OVERLY
COMPLICATED
THEORY?
Internal validity is best promoted via random assignment,an omnibus mechanism
that ensuresthat we do not have many assumptions to worry about when causal in-
ferenceis our goal. By contrast, quasi-experimentsrequire us to make explicit many
assumptions-the threats to internal validity-that we then have to rule out by fiat,
by design,or by measurement.The latter is a more complex and assumption-riddled
processthat is clearly inferior to random assignment.Something similar holds for
causal generalization,in which random selectionis the most parsimonious and the-
oretically justified method, requiring the fewest assumptionswhen causalgeneral-
ization is our goal. But becauserandom selectionis so rarely feasible,one instead
has to construct an acceptabletheory of generaliz tion out of purposive sampling,
'We
a much more difficult process. have tried to do this with our five principles of
generalizedcausal inference.These, we contend, are the keys to generalizedinfer-
ence that lie behind random sampling and that have to be identified, explicated,
ano assessed
and assessedifrt we are to make
make better general inferences, if they are not per-
rnterences,even rt
fect ones. But these principles are much more complex to implement than is ran-
dom sampling.
Let us briefly illustrate this with the category called American adult women.
We could represent this category by random selection from a critically appraised
register of all women who live in the United Statesand who arc at least 21 years
of age.I7ithin the limits of sampling error, we could formally generalizeany char-
acteristics we measured on this sample to the population on that register. Of
course, we cannot selectthis way becauseno such register exists.Instead,one does
onet experiment with an opportunistic sample of women. On inspection they all
'1,9
turn out to be between and 30 years of age, to be higher than average in
achievementand abilit5 and to be attending school-that is, we have useda group
of college women. Surface similarity suggeststhat each is an instance of the cate-
gory woman. But it is obvious that the modal American woman is clearly not a
college student. Such students constitute an overly homogeneoussample with re-
spect to educational abilities and achievement,socioeconomicstatus, occupation,
and all observable and unobservable correlates thereof, including health status,
current employment, and educational and occupational aspirations and expecta-
tions. To remedy this bias, we could use a more complex purposive sampling de-
sign that selectswomen heterogeneouslyon all these characteristics.But purpo-
sive sampling for heterogeneousinstances can never do this as well as random
selection can, and it is certainly more complex to conceive and execute.I7e could
go on and illustrate how the other principles faclhtate generalization. The point is
that any theory of generalization from purposive samples is bound to be more
complicated than the simplicity of random selection.
But becauserandom selection is rarely possible when testing causal relation-
ships within an experimental framework, we need these purposive alternatives.
I 499
NONEXPERIMENTALALTERNATIVES
Yet most experimental work probably still relies on the weakest of these alterna-
tives, surfaci similarity.'We seek to improve on such uncritical practice. Unfortu-
nately though, there is often restricted freedom for the more careful selection of
instancesof units, treatments, outcomes, and settings, even when the selection is
done purposively.It requires resourcesto sample irrelevanciesso that they are het-
erogeneouson many attributes, to measure several related constructs that can be
discriminated from each other conceptually and to measure a variety of possible
explanatory processes.This is partly why we expect more progress on causal gen-
eralization from a review context rather than from single studies. Thus, if one re-
searcher can work with college women, another can work with female school-
teachers, and another with female retirees, this creates an opportunity to see if
thesesourcesof irrelevant homogeneity make a difference to a causal relationship
or whether it holds over all these differ6nt types of women.
UltimatelS causal generalizationwill always be more complicated than assess-
ing the likelihood that a relationship is causal.The theory is more diffuse, more re-
cent, and lesswell testedin the crucible of researchexperience.And in some quar-
ters there is disdain for the issue,given the belief and practice that relationshipsthat
replicate once should be consideredas generaluntil proven otherwise' not to speak
oithe belief that little progressand prestigecan be achieved by designingthe next
experiment to be some minor variant on past studies. There is no point in pre-
t.nding that causal generalization is as institutionalized procedurally as other
methods in the social sciences.'Wehave tried to set the theoretical agendain a sys-
tematic way. But we do not expect to have the last word. There is still no explica-
tion of causal generalizationequivalent to the empirically produced list of threats
to internal validiry and the quasi-experimental designsthat have evolved over 40
years to rule out thesethreats. The agendais set but not complete.
RIM ENTALALTERNATIVES
NONEXPE
Though this book is about experimentalmethodsfor answeringquestionsabout
.".rr"l hypotheses,it is a mistaketo believethat only experimentalapproachesare
used for thir p,r.pose.In the following; we briefly consider severalother ap-
proaches,indiiating the major reasonswhy we havenot dwelt on them in detail.
basicallSthe reasonis that we believethat, whatevertheir merits for somere-
searchpurposes,they generatelessclearcausalconclusionsthan randomizedex-
perimentsor eventhe bestquasi-experiments or
suchas regression-discontinuity
interruptedtime series
The nonexperimentalalternativeswe examineare the major onesto emerge
in variousacademicdisciplines.In educationand parts of anthropologyand soci-
ologg one alternativeis intensivequalitativecasestudies.In thesesamefields,and
also-in developmentalpsychologythere is an emerginginterestin theory-based
500 14.A CR|T|CAL
ASSESSMENT
OFOURASSUMPTTONS
|
IntensiveQualitativeCaseStudies
The call to generate causal conclusions from intensive case studies comes from
several sources. One is from quantitative researchersin education who became
disenchanted with the tools of their trade and subsequently came to prefer the
qualitative methods of the historian and journalist and especiallyof the ethnog-
rapher (e.g.,Guba,198l, 1,990;and more tentatively Cronbach, 1986).Another
is from those researchersoriginally trained in primary disciplines such as qualita-
tive anthropology (e.g.,Fetterman, 19841or sociology (Patton, 1980).
The enthusiasm for case study methods arises for several different reasons.
One is that qualitative methods often reduce enough uncertainty about causation
to meet stakeholderneeds.Most advocatespoint out that journalists,historians,
ethnographers, and lay persons regularly make valid causal inferences using a
qualitative processthat combines reasoning, observation, and falsificationist pro-
cedures in order to rule out threats to internal validity-even if that kind of lan-
guage is not explicitly used (e.g.,Becker,1958; Cronbach,1982). A small minor-
ity of qualitative theorists go even further to claim that casestudiescan routinely
replace experiments for nearly any causal-sounding question they can conceive
(e.g.,Lincoln & Guba, 1985). A secondreasonis the belief that suchmethodscan
also engagea broad view of causation that permits getting at the many forces in
the world and human minds that together influence behavior in much more com-
plex ways than any experiment will uncover.And the third reasonis the belief that
case studies are broader than experiments in the types of information they yield.
For example, they can inform readers about such useful and diverse matters as
how pertinent problems were formulated by stakeholders, what the substantive
theories of the intervention are, how well implemented the intervention compo-
nents were, what distal, as well as proximal, effects have come about in respon-
dents' lives, what unanticipated side effects there have been, and what processes
explain the pattern of obtained results.The claim is that intensivecasestudy meth-
ods allow probes of an A to B connection, of a broad range of factors condition-
ing this relationship, and of a range of intervention-relevant questions that is
broader than the experiment allows.
.J
| 501
NONEXPERIMENTALALTERNATIVES
I
Evaluations
Theory-Based
This approach has beenformulated relatively recently and is describedin various
books or specialjournal issues(Chen & Rossi, 1,992;Connell, Kubisch, Schorr,&
'Weiss,
1.995;Rogers,Hacsi, Petrosino,& Huebner, 2000). Its origins are in path
analysis and causal modeling traditions that are much older. Although advocates
have some differenceswith each other, basically they all contend that it is useful:
(1) to explicate the theory of a treatment by detailing the expected relationships
among inputs, mediating pfocesses,and short- and long-term outcomes; (2) to
measure all the constructs specified in the theory; and (3) to analyzethe data to
assessthe extent to which the postulated relationships actually occurred. For
shorter time periods, the available data may addressonly the first part of a pos-
tulated causal chain; but over longer periods the complete model could be in-
volved. Thus, the priority is on highly specific substantive theorS high-quality
measurement,and valid analysisof multivariate explanatory processesas they un-
fold in time (Chen & Rossi, 1'987,1,992).
Such theoretical exploration is important. It can clarify general issueswith treat-
ments of a particular type, suggestspecific researchquestions,describehow the inter-
vention functions, spell out mediating processes,locate opportunities to remedy im-
plementation failures, and provide lively anecdotesfor reporting results ('Weiss,1'998).
All th.r. serveto increasethe knowledge yield, evenwhen such theoretical analysisis
done within an experimental framework. There is nothing about the approach that
makes it an alternative to experiments. It can clearly be a very important adjunct to
such studies,and in this role we heartily endorsethe approach (Cook,2000).
However, some authors (e.g., Chen 6c Rossi, 1,987, 1992; Connell et al.,
1,995l have advocated theory-based evaluation as an attractive alternative to ex-
periments when it comes to testing causal hypotheses.It is attractive for several
i.urorrr. First, it requires only a treatment group' not a comparison group whose
502 | 14.A CRTT|CAL
ASSESSMENT
OFOURASSUMPTTONS
Weaker Quasi-Experi
ments
For some researchers,random assignment is undesirable for practical or ethical
reasons, so they prefer quasi-experiments. Clearly, we support thoughtful use of
quasi-experimentation to study descriptive causal questions. Both interrupted
time series and regression discontinuity often yield excellent effect estimates.
Slightly weaker quasi-experiments can also yield defensible estimates,especially
when they involve control groups with careful matching on stable pretest attrib-
utes combined with other design features that have been thoughtfully chosen to
addresscontextually plausible threats to validity. However, when a researchercan
choose, randomized designsare usually superior to nonrandomized designs.
This is especially true of nonrandomized designs in which little thought is
given to such matters as the quality of the match when creating control groups,
j
I tOl
NONEXPERIMENTALALTERNATIVES
StatisticalControls
In this book,we haveadvocatedthat statisticaladjustmentsfor groupnonequivalence
are best urrd oBt designcontrolshavealreadybeenusedto the maximum in order
to reducenonequivalence to a minimum. So we are not opponentsof statisticalad-
justmenttechniquessuchasthoseadvocatedby the statisticiansand econometricians
describedin the appendixto Chapter5. Ratheqwe want to usethem as the last re-
sort.The positionwe do not like is the assumptionthat statisticalcontrolsare sowell
developeithat they can be usedto obtain confidentresultsin nonexperimentaland
weak iuasi-e*perimentalcontexts.As we saw in Chapter 5, researchin the past 2
504 | ta. a cRtTtcAL
AsSEssMENT
OFOURASSUMPT|ONS
I
decadeshas not much supported the notion that a control group can be constructed
through matchingfrom somenational or state registrywhen the treatmentgroup
comesfrom a morecircumscribedand localsetting.Nor hasresearchmuchsupported
the useof statisticaladjustmentsin longitudinalnationalsurveysin which individuals
with differentexperiences are explicitly contrastedin order to estimatethe effectsof
this experiencedifference.Undermatchingis a chronic problem here,as are conse-
quencesof unreliabilityin the selectionvariables,not to speakof specificationerrors
dueto incompleteknowledgeof the selectionprocess.In particular,endogeneity prob-
'We
lemsarea realconcern. areheartenedthat more recentwork on statisticaladjust-
mentsseemsto be moving toward the position we represent,with greateremphasis
beingplacedon internal controls,on stablematchingwithin suchinternalcontrols,
on the desirabilityof seekingcohort controlsthroughthe useof siblings,on the useof
pretests
PrstssLs collected
sorrccf,e(Jon
on the
rne same
same measures the posttest,
measures aS tne posttest, on
On the utiliw Ot
tne Uulrty of SUCh
suchpretest
measures collected at several different times, and on the desirability of studying inter-
'We
ventionsthat areclearlyexogenousshocksto someongoingsystem. arealsoheart-
enedby the progressbeingmadein the statisticaldomainbecause it includesprogress
on designconsiderations, aswell ason analysisper se(e.g.,Rosenbaum,1999a).Ve
areagnosticat this time asto the virtuesof the propensityscoreandinstrumentalvari-
able approachesthat predominatein discussionsof statisticaladiustmenr.Time will
tell how well
tell well they
they pan out relative to the results from randomizedexperiments.'We
have surely not heard the last word on this topic.
CONCLUSION
'We
cannot point to one new development that has revolutionized field experimen-
tation in the past few decades,yet we have seena very large number of incremen-
tal improvements. As a whole, these improvements allow us to create far better
field experiments than we could do 40 years ago when Campbell and Stanley
(1963) first wrote. In this sense,we are very optimistic about the future. Ve believe
that we will continue to see steadg incremental growth in our knowledge about
how to do better field experiments. The cost of this growth, howeveq is that field
experimentation has become a more specializedtopic, both in terms of knowledge
developmentand of the opportunity to put that knowledge into practice in the con-
duct of field experiments. As a result, nonspecialistswho wish to do a field exper-
iment may greatly benefit by consulting with those with the expertise,especiallyfor
large experiments, for experiments in which implementation problems may be
high, or for casesin which methodological vulnerabilities will greatly reducecred-
ibility. The same is true, of course, for many other methods. Case-studymethods,
for example, have become highly enough developed that most researcherswould
do an amateurishjob of using them without specializedtraining or supervisedprac-
tice. Such Balkanization of. methodolog)r is, perhaps, inevitable, though none the
lessregrettable.\U7ecan easethe regret somewhat by recognizingthatwith special-
ization may come faster progress in solving the problems of field experimentation.