See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/240801256

Multisite Qualitative Policy Research: Optimizing Description and

Generalizability

Article  in  Educational Researcher · February 1983

DOI: 10.3102/0013189X012002014

CITATIONS READS
479 3,090

2 authors, including:

William A. Firestone
Rutgers, The State University of New Jersey
119 PUBLICATIONS   4,734 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Systematic Review of How Teacher Evaluation Affects Teacher Data Use View project

All content following this page was uploaded by William A. Firestone on 06 November 2014.

The user has requested enhancement of the downloaded file.

 Multisite Qualitative Policy Research:
Optimizing Description and
Gene ral izab iI Y
ROBERTE. HERRIOTT and
Research Psychologist
The classical qualitative educa-
tional research design is the case
s t u d y . S t u d i e s of school life
(Cusick, 1973; Wolcott, 1973), of
the larger social forces affecting
schooling (Ogbu, 1974), and of ef-
forts to promote p l a n n e d educa-
tional change (Smith & Keith,
1971) have used qualitative data
in describing a single social set-
ting. Typically, such studies em-
phasize in-depth description but
provide a weak basis for general-
ization to other settings.
The last decade, however, has
seen the emergence of a new form
of qualitative research, one in-
tended to strengthen its ability to
generalize while preserving in-
depth description. These multisite
q u a l i t a t i v e s t u d i e s address the
same research question in a num-
ber of settings using similar data
collection and analysis procedures
in each setting. They consciously
seek to permit cross-site compari-
son w i t h o u t necessarily sacrific-
ing within-site understanding. Al-
though having some roots in aca-
demic social sciences (e.g., see
Clark, 1970; Whiting, 1963; Whit-
ing & Whiting, 1975), m u l t i s i t e
q u a l i t a t i v e r e s e a r c h arose pri-
m a r i l y in response to pressures
Robert E. Herriott is a research psy-
chologist, 85 Jennie Dugan Road,
Concord, MA 01742. Specialization:
Sociology of organizations; research
design and management. William A.
Firestone is Director of Field Studies,
Research for Better Schools, 444
North Third Street, Philadelphia, PA
19123. Specialization: Sociology of
education, Qualitative research, Edu-
cational change.
14
WILLIAMA. FIRESTONE
R e s e a r c h For B e t t e r S c h o o l s
from the federal government in
the 1970s for studies that could
overcome some of the weaknesses
of large quantitative evaluations
without being limited by the par-
ticularism of the single-site case
study. Like m a n y hybrids, it is to-
day quite robust. However, these
multisite qualitative studies were
typically expensive endeavors and
were done for specific policy pur-
poses, which the current federal
a d m i n i s t r a t i o n seems n e i t h e r to
value nor to feel it can afford.
There are two i m p o r t a n t rea-
sons for reflecting on the historical
development and potential utility
of multisite qualitative policy re-
search at this time. Although it is
unwelcomed by most social scien-
tists, the current hiatus in com-
missioning policy research at the
federal level provides researchers
and policymakers with an oppor-
t u n i t y to consider these issues in
some detail. Further, the field of
policy research has matured to the
point where such considerations
can be very f r u i t f u l . In r e c e n t
years qualitative researchers have
moved beyond the need to defend
the legitimacy of their craft in the
policy a r e n a (Rist, 1977; Smith,
1978; S t a k e , 1978). Moreover,
q u a n t i t a t i v e researchers are be-
ginning to acknowledge a role for
qualitative research in policy and
evaluation studies (Cronbach,
1982; H o a g l i n , L i g h t , McPeek,
Mosteller, & Stoto, 1982) and to
consider t h e proper b a l a n c e of
qualitative and quantitative tech-
niques (Cook & Reichardt, 1979;
Smith & Louis, 1982). In addition,
practitioners of multisite qualita-
tive policy research now exhibit
s u f f i c i e n t c o n f i d e n c e in t h e i r
craftsmanship to begin a process
of public self-criticism with an eye
to improving their methods (Fire-
stone & H e r r i o t t , 1982; Miles,
1979; Smith & Louis, 1982; Yin,
1981).
Efforts to e x a m i n e m u l t i s i t e
qualitative policy research suffer,
however, from the absence of de-
scriptive data about the field's sta-
tus and growth. While there are
useful first-person accounts of in-
dividual projects (e.g., see Fetter-
man, 1982) the field lacks system-
atic knowledge about a range of
studies. The sections that follow
offer a start in that direction. First
we review the context and concern
that led to the use of this innova-
tive design in the 1970s and de-
scribe briefly an early effort to em-
ploy it in educational research. We
then present the results of a for-
mal survey of 25 educational stud-
ies using this general design to
highlight some of its institutional
and methodological features. Fi-
nally we speculate about the fu-
ture of this design and suggest
ways in which it might be refined
by academically oriented educa-
tional researchers.
H i s t o r i c a l Context
The 1960s were a period of dra-
matic growth in federal funding of
e d u c a t i o n a l p r o g r a m s and the
1970s one of even greater growth
in t h e sponsorship of r e s e a r c h
about them (National Institute of
Education, 1976). I n i t i a l l y such
federally funded policy research
concentrated on the outcome of
these programs, but gradually an
expansion occurred from an exclu-
sive concentration on pupil effects
Educational Researcher
to an interest as well in program
implementation and its contexts.
Associated with this broadening
in research focus was a shift in
research designs from those that
were exclusively q u a n t i t a t i v e to
those that were a mixture of quan-
t i t a t i v e a n d q u a l i t a t i v e tech-
niques and even ones that were ex-
clusively q u a l i t a t i v e . A l t h o u g h
there was some advocacy in the
1970s for the use of better experi-
mental designs to capture illusive
program effects (see B e n n e t t &
L u m s d a i n e , 1975; R i e c k e n &
Boruch, 1974), the general thrust
of methodological innovation dur-
ing that decadeseems to have been
to better understand and measure
program i m p l e m e n t a t i o n . A key
component of this effort was the
use of qualitative data collection,
r e d u c t i o n , a n d a n a l y s i s tech-
niques.
The reasons for this emphasis on
qualitative designs have not been
well documented, but seem to in-
volve a mixture of scientific and
political considerations focused on
issues of political utility, design
validity, and forms clearance.
Prominent in the thinking of one
federal official was a concern that
the early evaluation designs were
"findings poor" in the sense that
they provided policymakers with
little understanding of why such
programs as Head Start and Foi-
low Through apparently had null
effects or of how to improve them
(Datta, 1982). A former federal of-
ficial argues that the displeasure
of policymakers with quantitative
studies arose less from a concern
about the lack of richness of their
findings as from a concern about
the findings themselves, ~few of
which were liked by program ad-
vocates." To him qualitative stud-
ies w e r e a t t r a c t i v e to policy-
m a k e r s because "case study ap-
proaches . . . t e n d to yield less
controversial findings, ones with
conclusions on both sides ofa polit-
ical decision" (Marshall Smith,
personal communication, 1982).
Regardless of where one stood
on the political utility ofquantita-
tive experimental studies there
also seems to have been disagree-
ment about the validity of their
key d e p e n d e n t v a r i a b l e s - - m e a -
sures of pupil p e r f o r m a n c e . To
February 1983
m a n y program advocates the then
existing standardized tests miti-
gated against the ability of even
the best q u a n t i t a t i v e designs to
show positive effects for those mi-
nority group members who were
the object of the most ambitious
federally funded educational ef-
forts (Cohen, 19751. Ethnographic
studies of minority children with-
in the complex cultural context of
their families, neighborhoods and
schools were often thought to be
more likely to document beneficial
program effects.
In a d d i t i o n to such concerns
about the validity of dependent
v a r i a b l e s were q u e s t i o n s a b o u t
that of the independent variables,
particularly the degree to which
programs implemented with fed-
eral funds were sufficiently faith-
ful to the intentions of their de-
signers, or enacted in a suffi-
c i e n t l y u n i f o r m m a n n e r across
most sites, to permit a meaningful
test of their effects (Weiss & Rein,
1970). One federal official respon-
sible for the design and implemen-
t a t i o n of t h e Follow T h r o u g h
planned variation experiments in-
c r e a s i n g l y came to favor more
qualitative studies which could as-
sist him and his colleagues in un-
derstanding why Follow Through
"models" were a p p a r e n t l y being
implemented so variously in dif-
ferent schools (McDaniels, person-
al communication).
A further contributor to the ex-
p a n s i o n f r o m q u a n t i t a t i v e to
qualitative policy studies seems to
have been the growth of a cumber-
some f o r m s - c l e a r a n c e process
w i t h i n the federal government.
Although introduced originally to
protect private i n d u s t r y from
obtrusive and r e d u n d a n t federal
data collection for regulatory pur-
poses, f o r m s c l e a r a n c e in t h e
1970s was embroiled in issues of
federal-state relations and of indi-
vidual privacy and became a major
obstacle to standardized data col-
lection efforts in educational re-
search (Datta, 1982). By the end of
the decade it was rare to see a
study of a federally funded educa-
tional program t h a t did not at
some point in its design rely on
u n s t a n d a r d i z e d d a t a collection
procedures, often to minimize (or
eliminate) the "forms clearance
hassle."
In sum, concerns about exces-
sive reliance on effects studies of
questionable utility and validity,
desires for more attention to pro-
gram implementation issues, and
efforts to avoid the "forms clear-
ance hassle" combined to create a
broad-based demand for policy re-
search modeled more after the tra-
ditional case study than after the
traditional experiment.
The R u r a l E x p e r i m e n t a l
Schools Study
The Rural Experimental
Schools Study was the first large-
scale, f e d e r a l l y f u n d e d educa-
tional research effort to explicitly
attempt the production of gener-
alizable findings using traditional
case study methods at multiple
sites. The federal official responsi-
ble for its g e n e r a l design was
openly skeptical of the utility of
exclusively quantitative methods
in program evaluation and argued
actively and successfully for a
strong qualitative component
(Budding, 1972). Abt Associates'
proposal to study the program's 10
rural sites adopted a two-pronged
approach to optimizing descrip-
tion and g e n e r a l i z a b i l i t y - - a se-
ries of quantitative cross-site stud-
ies were to be carried out simulta-
neously with 10 site-specific case
studies. In d e s c r i b i n g the case
study procedures, the project team
placed great emphasis on the use
of traditional ethnographic meth-
ods from the field of anthropology.
S u b s e q u e n t l y , experienced an-
thropologists and sociologists
were recruited by Abt Associates
to serve for approximately 3 years
as full-time, "on-site researchers"
at each rural site (Herriott, 1977,
1982).
The major product of these field
workers was to be a case study of
the site's educational change proj-
ect in the context of its school sys-
tem, community, and broader so-
ciocultural context. After consid-
erable negotiations among field
workers, the project's administra-
tion, and representatives of the
funding agency (the National In-
stitute of Education), it was agreed
that within broad guidelines, the
case study authors would be free to
let events at their sites be the pri-
mary determinants of case study
15
content (Fitzsimmons, 1975). In
addition the field workers were to
prepare detailed social and educa-
tional histories for each site, pro-
vide periodic qualitative reports
on local events to those at Abt As-
sociates' headquarters, and assist
in the collection of structured data
r e q u i r e d for t h e q u a n t i t a t i v e
cross-site studies.
The adoption of the case study
m e t h o d in its classical form al-
lowed these researchers to learn
a b o u t its s t r e n g t h s a n d weak-
nesses in the policy context and to
develop a d a p t a t i o n s to enhance
generalizability. The s t r e n g t h of
the Rural E x p e r i m e n t a l Schools
Study proved to be its ability to
generate in-depth description. The
major product of the qualitative
s e g m e n t of the study was eight
book-length case studies of change
projects (e.g., see Clinton, 1979;
Firestone, 1980).
When the Rural Experimental
Schools Study was commissioned
little was known about how to use
qualitative methodologies to de-
velop general conclusions and ini-
t i a l l y less effort was devoted to
that task t h a n to the production of
site-specific narratives. Over time,
however, two important steps were
made in the direction of gener-
alizability. One involved the incur-
poration of qualitative data into
studies t h a t were originally in-
tended to be solely q u a n t i t a t i v e
cross-site comparisons. Thus the
cross-site survey of organizational
c h a n g e processes used r e p o r t s
from the on-site researchers as a
major source of data for its key de-
p e n d e n t v a r i a b l e - - t h e degree of
c o m p r e h e n s i v e c h a n g e at each
s i t e - a s well as for insights about
the nature of the change process in
specific locations (Rosenblum &
Louis, 1981). E q u a l l y i m p o r t a n t
were efforts to synthesize insights
from the case studies themselves
to m a k e g e n e r a l i z a t i o n s a b o u t
change processes in rural school
districts. Limited by the decision
to give the field workers autonomy
with respect to the content of his or
her study, these syntheses were
carried out in two ways. One ap-
proach had the field workers write
chapter-length case study narra-
tives to a common outline and then
asked outside e x p e r t s to draw
16
cross-case conclusions for policy-
makers and school administrators
(Herriott & Gross, 1979). The sec-
ond p r e p a r e d more t r a d i t i o n a l
' q i t e r a t u r e reviews" of the eight
completed book-length case stud-
ies for p o l i c y m a k e r s (Herriott,
1980) and school a d m i n i s t r a t o r s
(Deal & Nutt, 1979).
A Survey of Multisite
Qualitative Studies
The Rural Experimental
Schools Study was part of the ex-
pansion in the use of multisite
qualitative methods in the 1970s.
To learn more about this trend we
undertook a survey of federal offi-
cials and qualitatively oriented re-
searchers. T h r o u g h a snowball
samplil~g process we identified 24
other projects which: (1) were fed-
erally funded via a competitive
RFP process, (2) involved the ap-
plication of qualitative methods of
data collection within at least a
major part of the overall design,
and (3) intended to compare two or
more organizationally based re-
search sites.1 One of the most no-
ticeable features of these projects
is t h a t whereas single-site case
studies arise almost exclusively
w i t h i n a c a d e m i a most of these
m u l t i s i t e projects were located
within the type of applied social
research firm, which at that time
was specializing in quantitative
policy research. Overwhelmingly
the qualitative studies were em-
bedded within larger (multistudy)
projects having quantitative com-
ponents as well, and thus provided
opportunities not only for cross-
site qualitative synthesis, but for
the integration of qualitative and
quantitative data. The funding for
these projects was rather extensive
(typically over 1 million dollars)
and their duration lengthy (typ-
ically at least two years). 2
Four Design Issues
While the i n t e n t of multisite
qualitative policy research is to
optimize description and gener-
alizability, there is a persistent
tension between these two objec-
tives that permeates all research
(Cook & C a m p b e l l , 1979;
M c G r a t h , 1982). In m u l t i s i t e
qualitative research this tension
seems to revolve around four de-
sign issues. The most prominent of
these issues is the degree to which
the data collection effort should be
' % t r u c t u r e d " ( F i r e s t o n e & Her-
riott, 1982). Cross-site comparison
and g e n e r a l i z a t i o n r e q u i r e re-
searchers at all sites to use shared
definitions of concepts and com-
mon data collection procedures to
ensure that cross-site similarities
and differences are characteristics
of the sites and not the result of
m e a s u r e m e n t procedures or re-
searcher bias (Pelto & Pelto, 1978).
Yet such standardization encour-
ages r e s e a r c h e r s to ignore the
unique aspects of each site and to
overlook processes and contexts
that may make special contribu-
tions to the phenomena of interest.
They also encourage the research-
ers to impose their definitions of
the situation through premature
conceptualization (Blumer, 1969).
A high degree of structuring of
data collection is obtained through
the use of closed-end, precoded
questionnaires and interview
schedules. Unstructured modes of
data collection include unobtru-
sive observation and schedule-free
interviewing. These are the pri-
mary forms of data collection for
most traditional case studies. Our
snowball s a m p l i n g process ex-
cluded projects t h a t relied pri-
marily on highly structured data
collection. Nevertheless, when we
examined the data collection pro-
c e d u r e s e m p l o y e d by a m a j o r
qualitative study w i t h i n each of
these 25 projects, we were sur-
prised to find that only five relied
p r i m a r i l y on u n s t r u c t u r e d data
collection techniques. The other
20 employed primarily a variety of
s e m i - s t r u c t u r e d procedures, in-
cluding site-visit guides, which
specify the questions that must be
answered but not the specific data
sources to be used, open-end inter-
view guides, and instructions for
focused observation. Such meth-
ods require that research issues be
well thought out in advance rather
t h a n being derived "in the field."
The heavy reliance on semi-struc-
tured procedures is clearly a major
d e p a r t u r e from the t r a d i t i o n a l ,
single-site case study approach. It
seems to represent an accomoda-
tion in the direction of quantita-
Fducational Researcher
tive methods, one made to facili-
tate cross-site comparison.
A second design issue concerns
the n u m b e r of sites to be studied.
To a point, generalizability is en-
hanced by the inclusion of m a n y
sites (Kennedy, 1979). However, for
any given budget level, increasing
the n u m b e r of sites limits the re-
sources th at are available for de-
scribing and analyzing events at
any one site or for cross-site com-
parison. Within this sample, the
fewest sites studied were three and
the most were 60 with a median of
11. The 25 studies seem to cluster
into th r ee distinct groups: those
with three th r o u g h six sites (7 in-
stances), those with eight through
22 sites (13 instances), and those
with 30 t h r o u g h 60 sites (5 in-
stances). The five studies with over
30 sites raise an interesting ques-
tion: how does one synthesize the
mass of qualitative data from so
m a n y locations when attempting
to draw generalizations? One risk
in a t t e m p t i n g such a cross-site
analysis is that the analyst will
draw on the sites selectively, thus
reducing data complexity but at
the expense of representativeness.
One alternative to such selectivity
is to quantify the qualitative data
t h ro u g h the use of rigorous coding
schemes so that formal statistical
models can be used in carrying out
the cross-site analysis. Yet such
quantification can u nder m i ne the
descriptive value of qualitative re-
search th at the multisite design is
intended to exploit.
A third issue is the length of
time to be spent at each site for
purposes of data collection. Long-
t e rm immersion (generally of over
one year) is the h a l l m a r k of clas-
sical eth n o g r ap h y (Wolcott, 1975)
and is an important means of en-
suring valid description (Dawson,
1982). However, i n c r e a s i n g t h e
a m o u n t of time at any one site lim-
its t h e r e s o u r c e s a v a i l a b l e for
studying other sites and for cross-
site comparison and generaliza-
tion. On-site presence in this sam-
ple of 25 studies fell into t h r e e
broad categories: one or two short
visits to each site (10 instances),
several i n t e r m i t t e n t visits (7 in-
stances), a n d m o r e c o n t i n u o u s
field work (8 instances).
Finally, the research team can
February 1983
emphasize site-specific reporting,
as was done in the Rural Experi-
m e n t a l Schools Study, or cross-
site, issue-specific reporting. Site-
specific reporting is a literary de-
vice t ha t enhances description but
tends to mask similarities and dif-
ferences across sites, thereby in-
h i b i t i n g g e n e r a l i z a t i o n . Cross-
site, issue-specific reporting facili-
tates generalization, but often at
t he expense of site-specific con-
text. Although most of the 25 stud-
ies we surveyed used both site-spe-
cific and cross-site qualitative re-
p o r t i n g formats, 12 em phasi zed
the former and 13 the latter.
Interissue Patterns
In c o n d u c t i n g t h i s s u r v e y of
m u l t i s i t e q u a l i t a t i v e policy re-
search projects we sought to un-
cover possible patterns in the de-
gree to which decisions affecting
one of these four design issues are
associated with decisions on the
o t h e r t h r e e . For e x a m p l e , it
seemed that the n u m b e r of sites
could well be an i m port ant factor
in d e t e r m i n i n g t h e d e g r e e to
which r e p o r t n a r r a t i v e s empha-
size site-specific p h e n o m e n a , for
there might be a point at which
site-specific narrative becomes too
cumbersome to utilize across all
sites. Such a p a t t e r n does exist
across these 25 projects. Whereas
six of the seven projects having six
or fewer sites made extensive use
'of site-specific narrative, none of
the 5 projects with 30 or more sites
did so. However, when we looked
for associations between gross cat-
egories of the other five pairs of the
four design decisions we could find
none, suggesting little pervasive
pa t t e r n in how the n u m b e r of sites,
the degree of on-site presence, the
degree of em phasi s on u n s t r u c -
tured data collection, and the de-
gree of e m phasi s on w i t h i n - s i t e
narrative covary.3
T h e a b s e n c e of p a t t e r n i n g
a m ong all four design variables
could simply reflect our inability
to properly conceptualize or mea-
sure the most important choices
faced by t h o s e r e s p o n s i b l e for
these 25 studies, but we are in-
clined to view it as reflecting two
other considerations. In particular
it is i m po rt ant to keep in mind the
very recent origin of this research
form. During the past decade the
logical connections among its vari-
ous design options have been influ-
enced by considerable experimen-
tation on the part of both federal
officials and r e s e a r c h e r s , as to-
gether they endeavored to under-
s t a n d b e t t e r its s t r e n g t h s a n d
weaknesses. However, it must also
be kept in mind that such experi-
mentation has t aken place within
a context in which ~the logic of
science must come to terms with
t he logic of politics" (Cronbach,
1982, p. ix). Any policy research
project arises within a unique po-
litical environment. Further, each
project is modified over time to re-
flect changes in that environment
as they are interpreted by various
federal officials and social scien-
tists engaged in its design and im-
plementation. Although policy re-
s e a r c h e r s are clearly influenced
by the logic of science, their design
and implementation decisions are
not simply the result of the appli-
cation of that logic to explicit a
priori policy q u e s t i o n s . R a t h e r ,
their decisions result from the in-
teraction of political and scientific
considerations from the time of the
earliest conception of the need for
research through the completion
of the final report document. As a
result no complex pattern of design
and i m p l e m e n t a t i o n choices can
best satisfy the political require-
ments of more t han a single proj-
ect.
The Future of Multisite
Qualitative Research
Multisite q u a l i t a t i v e r e s e a r c h
fl ouri shed in t he federal policy
arena of the 1970s. It grew out of
the qualitative traditions of aca-
demic social science and was modi-
fled to take into account the larger
scale and more diverse audience of
the policy arena. U n f o r t u n a t e l y ,
its future utility at the federal or
state level or in academ i a is at
p r e s e n t unclear. At the federal
level, the future of this research
approach is enmeshed in debates
about the government's responsi-
bility for social action programs
and for research. All 25 projects we
surveyed were initiated to deter-
mine how the federal government
can intervene more effectively to
17
i m p r o v e s e r v i c e d e l i v e r y at t h e
s t a t e or local level. While the cur-
r e n t a d m i n i s t r a t i o n c l e a r l y es-
chews such a c t i v i s m at the federal
level as a m a t t e r of policy, it re-
m a i n s to be seen w h e t h e r the fed-
e r a l g o v e r n m e n t can fully divest
itself of this responsibility. To the
e x t e n t t h a t it c a n n o t - - o r to the ex-
t e n t t h a t a n a d m i n i s t r a t i o n ori-
e n t e d to s t a t e a n d local c o n t r o l
chooses to fulfill its responsibili-
ties by conducting research and
d e m o n s t r a t i o n p r o g r a m s to pro-
vide k n o w l e d g e w i t h o u t i m p o s i n g
m a n d a t e s for a c t i o n - - t h e u s e of
m u l t i s i t e q u a l i t a t i v e studies will
continue. Also likely to facilitate
t h e f u r t h e r use of t h i s innovative
m e t h o d a r e s o p h i s t i c a t e d policy
r e s e a r c h efforts at the s t a t e level.
The e v a l u a t i o n of the C a l i f o r n i a
School I m p r o v e m e n t P r o g r a m , for
e x a m p l e , involved extensive field
w o r k in 24 schools in 14 districts
( B e r m a n , Weiler, Czesak, Gjelten,
& Izu, 1981).
The c u r r e n t h i a t u s in the com-
m i s s i o n i n g of large-scale policy re-
s e a r c h provides a c a d e m i c a l l y ori-
ented educational researchers
w i t h u n u s u a l o p p o r t u n i t i e s to ex-
p e r i m e n t w i t h this design. In the
p r o c e s s m u l t i s i t e q u a l i t a t i v e re-
s e a r c h will h a v e to be a d a p t e d to
t h e a c a d e m i c s e t t i n g which, un-
like the federal policy context,
generally requires that research
be done at m o r e m o d e s t cost b u t
w i t h longer t i m e lines.
A c a d e m i c r e s e a r c h e r s can facili-
t a t e t h e d e v e l o p m e n t of m u l t i s i t e
qualitative research by address-
ing a b r o a d r a n g e of methodologi-
cal issues. For example, one cur-
r e n t n e e d is to u n d e r s t a n d b e t t e r
the c o n s e q u e n c e s of different staff-
ing p a t t e r n s for d a t a collection.
T h i s is a n especially i m p o r t a n t is-
sue in q u a l i t a t i v e r e s e a r c h w h e r e
t h e r e s e a r c h e r is often t h e crucial
" i n s t r u m e n t " ( S a n d a y , 1979). It
m a y be useful to conduct r e s e a r c h
that compares alternative data
collection p a t t e r n s . One such pat-
t e r n is t h e use of a single investi-
g a t o r to c a r r y out all field w o r k in
all sites (see Metz, 1978). Such an
approach standardizes the data
collection ~instrument" across
sites w i t h o u t sacrificing the poten-
tial for i n - d e p t h description, b u t it
s e e m s l i m i t e d to s i t u a t i o n s involv-
18
i n g no m o r e t h a n t h r e e or four
sites. An a l t e r n a t i v e possibility is
to provide g r e a t e r d a t a collection
s t r u c t u r e across m u l t i p l e field
w o r k e r s , e i t h e r t h r o u g h the use of
field manuals (Campbell & type i l l u s t r a t e d above can seldom
Levine, 1973) or by h a v i n g the dif-
f e r e n t field w o r k e r s p r e p a r e case
s t u d y n a r r a t i v e s for t h e i r s i t e s
u s i n g a c o m m o n f o r m a t a g r e e d to
a f t e r c o n d u c t i n g some field w o r k
(see H e r r i o t t & Gross, 1979). AI-
t h o u g h t h e r e h a s b e e n some effort
to c o m p a r e such approaches, it h a s
not b e e n as s y s t e m a t i c or as exten-
sive as it could be ( P e r l m a n , 1973).
It would also be useful to k n o w the
conditions u n d e r which it is pref-
e r a b l e to use "local" r e s i d e n t s or
p r o f e s s i o n a l r e s e a r c h e r s as field
observers and about the advan-
t a g e s a n d d i s a d v a n t a g e s of doing
cross-site c o m p a r i s o n and gener-
a l i z a t i o n w i t h field w o r k e r s col-
l a b o r a t i n g "in c o m m i t t e e " or with
~'outside e x p e r t s " who w o r k only
w i t h t h e s i t e - s p e c i f i c case s t u d y
narratives.
A n o t h e r crucial issue is the con-
s e q u e n c e s of different a p p r o a c h e s
to t h e s t a n d a r d i z e d r e d u c t i o n of
u n s t a n d a r d i z e d data. Such reduc-
tion is a n e c e s s a r y first step to a n y
a n a l y s i s w i t h i n or a c r o s s s i t e s
(Goetz & L e C o m p t e , 1981). T h e po-
t e n t i a l of a n y s t u d y for useful, val-
id description a n d g e n e r a l i z a t i o n
depends on the a n a l y s t s ' ability to
reduce d a t a to a m a n a g e a b l e f o r m
w i t h o u t distortion or loss of m e a n -
ingful detail. Studies w i t h a l a r g e
n u m b e r of sites, or w h e r e the prin-
cipal i n v e s t i g a t o r is not i n t i m a t e l y
f a m i l i a r w i t h all locations, are es-
p e c i a l l y d e p e n d e n t on t h e i r ap-
p r o a c h e s to d a t a reduction. While
we c u r r e n t l y h a v e some craft dis-
cussion of how d a t a r e d u c t i o n was
done in specific projects, we need
to k n o w m o r e a b o u t t h e a d v a n -
t a g e s a n d d i s a d v a n t a g e s of t h e
q u a n t i f i c a t i o n of q u a l i t a t i v e d a t a
( s e e L o u i s , 1982; T a l m a g e &
Rasher, 1981), a n d of v e r b a l t a b u -
l a r a n d g r a p h i c d a t a reduction de-
v i c e s (see H u b e r m a n & M i l e s ,
1983; S m i t h & N e r e n b e r g , 1981).
O t h e r issues in need of a t t e n t i o n
a r e t h e t i m i n g of site visits in light
of t h e p h e n o m e n a u n d e r study, ex-
a m i n a t i o n of p r o c e s s e s a n d out-
c o m e s at d i f f e r e n t p r o g r a m a t i c
levels (student, classroom, school,
district, etc.) and alternative
m o d e s of p r e s e n t i n g the r e s u l t s of
r e s e a r c h to p o l i c y m a k e r s .
Due to the p r e s s u r e of time, m a -
j o r m e t h o d o l o g i c a l i s s u e s of t h e
be a d d r e s s e d s y s t e m a t i c a l l y in the
course of policy studies. A c a d e m -
ically oriented methodological
studies r e p r e s e n t an o p p o r t u n i t y
to m o r e fully explicate t h e logic of
this developing r e s e a r c h f o r m and
to e x a m i n e in d e t a i l i t s a p -
plicability in various policy con-
texts.
Footnotes
This paper has been prepared with
support from The National Institute of
Education under contract No. 400-80-
0019. It does not, however, neces-
sarily reflect the view of that agency.
We are particularly indebted to Fritz
Mulhauser of the Institute's staff for
his u n f a i l i n g f a c i l i t a t i o n of our re-
search.
1The snowball sampling process began
with several highly visible qualitative re-
searchers IKaren S. Louis, Matthew B.
Miles, Ray C. Rist, Robert Yin) and federal
officials (Edward Glassmam Frederick
Mulhause~ Marshall Smith, James Van-
eckoi. Through their recommendations--
and the recommendations of persons sug-
gested by them--a roster of approximately
100 candidate projects was created. Subse-
quent telephone calls to a person more
knowledgeable about each project led to the
elimination of approximately 75 projects,
in most cases due to a failure to satisfy all
three of the sampling criteria. For those
projects meeting all criteria, arrangements
were made for a 1-hour telephone inter-
view~ generally with the project's director.
At the time of the interview the informant
was queried about his or her project using a
highly structured "project profile" sheet as
a guide. After the interview was finished a
draft copy of the complete profile was sent
to the informant and modifications re-
quested if necessary. After the full set of 25
profiles had been created and reviewed,
they were used to code each project in terms
of a series of summary categories. The key
informants then reviewed that coding and
suggested whatever further modification of
the profile sheets or summary tables
seemed warranted.
2For a detailed description of each of the
25 projects, see Herriott & Firestone (1982,
Appendix A).
aFor the basic tables supporting these
conclusions, see Herriott & Firestone
(1982, Appendix C).
Educational Researcher
 References
Bennett, C.A., & Lunsdaine, A.A. (Eds.).
Evaluation and experiment: Some erit-
ical issues in assessing social programs.
New York: Academic Press, 1975.
Berman, P., Weilet, D., Czesak, K., Gjelten,
T., & Izu, J. A. How schools view and use
the School hnprovement Program: Pre-
liminary hypotheses. Berkeley, Calif.:
Manifest International, 1981.
Blumer, H. Symbolic interactionism: Per-
spective and method. Englewood Cliffs,
N.J.: Prentice-Hall, 1969.
Budding, D. Draft discussion paper on Ex-
perimental Schools Program. Wash-
ington, D.C.: Experimental Schools Pro-
gram, 1972.
Campbell, D.T., & Levine, R.A. Field-man-
ual anthropology. In R. Naroll & R. Co-
hen (Eds.), A handbook of method in cul-
tural anthropology. New York: Columbia
University Press, 1973.
Clark, B.R. The distinctive college: Anti-
och. Reed & Swathmore. Chicago: AI-
dine, 1970.
Clinton, C.A. Local success and federal
failure: A study of cornmunity develop-
meat and educational change in the rural
South. Cambridge, Mass.: Abt Books,
1979.
Cohen, D.K. The value of social experi-
ments. In A.M. Rivlin & P.M. Timpane
(Eds.), Planned variation in education:
Should we give up or try harder? Wash-
ington, D.C.: Brookings I n s t i t u t i o n ,
1975.
Cook, T.D., & Campbell, D.T. Quasi-experi-
mental design: Design and analysis is-
sues for rield settings. Chicago, Ill.: Rand-
McNally, 1979.
Cook, T.D., & R e i c h a r d t , C.S. (Eds. I,
Qualitative and quantitative methods in
evaluation research. Beverly Hills, Calif.:
Sage, 1979.
Cronbach, L.J. Designing evaluations of ed-
ucational and social programs. San
Francisco: Jossey-Bass, 1982.
Cusick, P.A. Inside high school: The stu-
dent~ world. New York: Holt, Rinehart &
Winston, 1973.
Datta, L. Strange bedfellows: The politics
of qualitative methods. American Behav-
ioral Scientist, 1982, 26(1), 133-144.
Dawson, J.A. Qualitative research findings:
What do we do to improve and estimate
their validity ? Paper presented at the an-
nual meeting of the American Educa-
tional Research Association, March
1982.
Deal, T.E., & Nutt, S.C. Promoting, guiding
and surviving change in school districts.
Cambridge, Mass.: Abt Associates Inc.,
1979.
Fetterman, D.M. Ethnography in educa-
tional research: The dynamics of diffu-
sion. Educational Researcher, 1982,
11(3), 17-22, 29.
Firestone, W.A. Great expectations /br
small schools: The limitations of federal
projects. New York: Praeger, 1980.
Firestone, W.A., & Herriott, R.E., The for-
malization of qualitative research: An
February 1983
View publication stats
adaptation of "soft" science to the policy
world. In R.E. Herriott & W.A. Firestone
(Eds.), Multisite qualitative policy' re-
search in education: A study o/'reeent fed
eral experience. Concord, Mass.: Authors,
1982. IFinal Report No. NIE-400-80-
0019).
Fitzsimmons, S.J. The anthropologist in a
s t r a n g e land. H u m a n Organization,
1975, 34, 183-196.
Goetz, J.P., & LeCompte, M.D. Eth-
nographic research and the problem of
data reduction. Anthropology and Edu-
cation Quarterly, 1981, 12(li, 51 70.
Herriott, R.E. Ethnographic case studies in
federally-funded multi-disciplinary pol-
icy research: Some design and imple-
mentation issues. Anthropology and Ed-
ucation Quarterly, 1977, 9t2i, 106-115.
Herriott, R.E. Federal initiatives and rural
school improvement: Findings from the
Experimental Schools Program. Cam-
bridge, Mass.: Abt Associates Inc., 1980.
Herriott, R.E. Tensions in research design
and implementation: The Rural Experi-
mental Schools Study. American Behav-
ioral Scientist, 1982, 26(1), 23~14.
Herriott, R.E., & Firestone, W.A. ¢Eds.i.
Multisite qualitative policy research in
education: A study' of recent federal expe-
rience. Concord, Mass.: Authors, 1982.
(Final Report No. NIE-400-80-0019).
Herriott, R.E., & Gross, N. (Eds.i. The dy-
namics of planned educational change:
Case studies and analyses. Berkeley, Cal-
if.: McCutchan, 1979.
Hoaglin, D.C., Light, R.J., McPeek, B.,
Mostellel; F., & Stoto, M.A. Data for deci-
sions: Information strategies/'or policy,-
makers. Cambridge, Mass.: Abt Books,
1982.
Huberman, A.M., & Miles, M.B. Drawing
valid meaning from qualitative data:
Some techniques of data reduction and
display. Quality' and Quantity', 1983, 17.
Kennedy, M.M. Generalizing from single
case studies. Evaluation Quarterly, 1979,
3, 661-678.
Louis, K.S. Sociologist as sleuth: Integrat-
ing methods in the RDU Study. Ameri-
can Behavioral Scientist, 1982, 26(1),
101-120.
McDaniels, G.L. Telephone interview with
Robert E. Herriott, July 27, 1982.
McGrath, J.E. Dilematics: The study of re-
search choices and dilemmas. In J.E.
McGrath, J. Martin, & R. Kulka (Eds.),
J u d g m e n t calls in research. Beverly
Hills, Calif.: Sage, 1982.
Metz, M.H. Classrooms and corridors: The
crisis of authority in desegregated sec-
ondary schools. Berkeley, Calif.: Univer-
sity of California Press, 1978.
Miles, M.B. Qualitative data as an attrac-
tire nuisance: The problem of analysis.
Administrative Science Quarterly, 1979,
24, 590-601.
National Institute of Education. 1976 data
book: The status of educational research
and development in the United States.
Washington, D.C.: Author, 1976.
Ogbu, J.U. The next generation: An eth-
nography of education in an urban neigh-
borhood. New York: Academic Press,
1974.
Pelto, P.J., & Pelto, G.H. Anthropological
research: The structure of inquiry t2nd
ed.). Cambridge: Cambridge University
Press, 1978.
Perlman, M.L. The comparative method:
The single investigator and the team ap-
proach. In R. Naroll & R. Cohen (Eds.), A
handbook of method in cultural an-
thropology. New York: Columbia Univer-
sity Press, 1973.
Riecken, H.W. & Boruch, R.F. (Eds.). Social
experimentation: A method for planning
and evaluating social intervention. New
York: Academic Press, 1974.
Rist, R. On the relations among educa-
tional research paradigms: From disdain
to detente. Anthropology and Education
Quarterly,, 1977, 8(2), 4 2 4 9 .
Rosenblum, S., & Louis, K.S. Stability and
change: Innovation in an educational
context. New York: Plenum, 1981.
Sanday, P.P. The ethnographic paradigm{ s).
Administrative Science Quarterly, 1979,
24, 527-539.
Smith, A.G., & Louis, K.S. (Eds.i. Multi-
method policy research: Issues and appli-
cations. American Behavioral Scientist,
1982, 26(1~, 1-144.
Smith, A.G., & Nerenberg, S. Parents and
federal educational programs, Vol. 5: Fol-
low Through. Santa Monica, Calif.: Sys-
tem Development Corporation, 1981.
Smith, L.M. An evolving logic of partici-
pant observation, educational ethnogra-
phy and other case studies. In L. Shul-
man (Ed.). Review of research in educa-
tion tVol. 6). Itasca, Ill.: Peacock, 1978.
Smith, L.M., & Keith, P.M. Anatomy ofedu-
cational innovation: A n organizational
analysis of an elementary school. New
York: Wiley, 1971.
Smith, Marshall S. Letter to Robert E. Her-
riott, June 28, 1982.
Stake, R.E. The case study method in social
inquiry. Educational Researcher, 1978,
7(2), 5-8.
Talmage, H., & Rasher, S.P. Quantifying
qualitative data: The best of both worlds.
Paper presented at the annual meeting of
the American Educational Research As-
sociation, Los Angeles, 1981.
Weiss, R.S., & Rein, M. The evaluation of
broad-aim programs: Experimental de-
sign, its difficulties, and an alternative.
Administrative Science Quarterly, 1970,
15, 97-109.
Whiting, B.B. (Ed.). Six cultures: Studies of
child rearing. New York: Wiley, 1963.
Whiting, B.B., & Whiting, J,W.M. Children
of six cultures: A psycho-cultural analy-
sis. Cambridge, Mass.: Harvard Univer-
sity Press, 1975.
Wolcott, H.F. The man in the principal's of-
rice: A n ethnography. New York: Holt,
Rinehart & Winston, 1973.
Wolcott, H.F. Criteria for an ethnographic
approach to research in schools. Human
Organization, 1975, 34, 111-128.
Yin, R.K, The case study crisis: Some an-
swers. Administrative Science Quarterly,
1981, 26, 58-65.
19