Evaluation and Program Planning 48 (2015) 137–148

Contents lists available at ScienceDirect

Evaluation and Program Planning

journal homepage: www.elsevier.com/locate/evalprogplan

‘‘They Just Know’’: The epistemological politics of ‘‘evidence-based’’

non-formal education
Thomas Archibald a,b,*
a
Cornell University, 3M14 Martha Van Rensselaer Hall, Ithaca, NY 14853, USA
b
Virginia Tech, Agricultural & Extension Education (0343), 276 Litton-Reaves Hall, 175 West Campus Drive, Blacksburg, VA 24061, USA

A R T I C L E I N F O A B S T R A C T

Community education and outreach programs should be evidence-based. This dictum seems at once
warranted, welcome, and slightly platitudinous. However, the ‘‘evidence-based’’ movement’s more
Article history:
Available online 17 August 2014 narrow definition of evidence—privileging randomized controlled trials as the ‘‘gold standard’’—has
fomented much debate. Such debate, though insightful, often lacks grounding in actual practice. To
address that lack, the purpose of the study presented in this paper was to examine what actually
Keywords:
Evidence-based programs happens, in practice, when people support the implementation of evidence-based programs (EBPs) or
Evidence-based practice engage in related efforts to make non-formal education more ‘‘evidence-based.’’ Focusing on three
Epistemology cases—two adolescent sexual health projects (one in the United States and one in Kenya) and one more
Epistemological politics general youth development organization—I used qualitative methods to address the questions: (1) How
Non-formal education is evidence-based program and evidence-based practice work actually practiced? (2) What perspectives
Adolescent sexual health and assumptions about what non-formal education is are manifested through that work? and (3) What
Randomized controlled trials conflicts and tensions emerge through that work related to those perspectives and assumptions?
Informed by theoretical perspectives on the intersection of science, expertise, and democracy, I conclude
that the current dominant approach to making non-formal education more evidence-based by way of
EBPs is seriously flawed.
ß 2014 Elsevier Ltd. All rights reserved.

1. Introduction and treatment work, the domain in which one of this paper’s focal
cases is active:
Community education and outreach programs should be based
In the context of research, treatment, and prevention, evidence
on evidence. On the face of it, this dictum seems at once warranted,
usually refers to qualitative and/or quantitative results that
welcome, and slightly platitudinous. Few people would argue that
have been published in a peer-reviewed journal. The term
an educational initiative should, or even could, be planned and
‘evidence-informed’ is preferred to ‘evidence-based’ in recog-
implemented in a way that is somehow devoid of evidence. Yet, as
nition of the fact that several elements may play a role in
the contributions to this volume suggest, what counts as credible
decision-making, only one of which may be scientific evidence.
evidence in evaluation and social research is far from self-evident.
Other elements may include cultural appropriateness, concerns
Relatedly, how such evidence is expected and understood to guide
about equity and human rights, feasibility, opportunity costs,
professional practice remains unclear, posing a daunting challenge
etc. (UNAIDS, 2011, pp. 10–11)
to policy-makers, researchers, evaluators, and educators alike. This
These terminology guidelines make explicit the perspective that
challenge is encapsulated in the ‘‘preferred terminology guide-
‘‘scientific evidence,’’ though important, is just one of many factors
lines’’ put forward by UNAIDS (the Joint United Nations Program on
to consider when creating or choosing between various educational
HIV and AIDS) to guide the discourse of global HIV/AIDS prevention
programs or practices. The guidelines presuppose that there exists
no single, universally accepted definition of ‘‘evidence,’’ a presup-
position of central importance to this paper. What’s more, the
* Correspondence to: Virginia Tech, Agricultural & Extension Education (0343),
statement glosses over decades of acerbic debates about the relative
276 Litton-Reaves Hall, 175 West Campus Drive, Blacksburg, VA 24061, USA.
Tel.: +1 540 231 6192. merits of quantitative versus qualitative evidence, a noteworthy
E-mail address: tgarch@vt.edu point because most dominant approaches to ‘‘evidence-based’’

http://dx.doi.org/10.1016/j.evalprogplan.2014.08.001
0149-7189/ß 2014 Elsevier Ltd. All rights reserved.
138 T. Archibald / Evaluation and Program Planning 48 (2015) 137–148
education value only quantitative evidence gathered through
‘‘rigorous’’ research or evaluation designs. Also, the statement
equates ‘‘evidence’’ with that which has been published in a peer-
reviewed journal. While many—especially scientists—might take
that aspect of the definition of evidence to be self-evidently correct,
many others—including scientists—work, pragmatically, with much
broader definitions of ‘‘evidence.’’
However, the ‘‘evidence-based’’ education movement—one of
many related attempts to ‘‘bridge the research-practice gap’’ that
has gained prevalence in recent decades—is predicated on more
formal and conscribed definitions of evidence, whereby certain
research and evaluation approaches are valued more highly than
others. In the current era of accountability, some policy-makers,
funding agencies, and scholars position ‘‘scientific’’ evidence
derived from randomized controlled trials (RCTs) as the ‘‘gold
standard’’ for establishing proof of which programs ‘‘work’’ and
which do not (Coalition for Evidence-Based Policy, 2003; Mosteller
& Boruch, 2002; U.S. Department of Education, 2003). According to
Trochim:
The gold standard debate is one of the most important
controversies in contemporary evaluation and applied social
sciences. It’s at the heart of how we go about trying to
understand the world around us. It is integrally related to what
we think science is and how it relates to practice. There is a lot at
stake. (W. Trochim, unpublished speech transcript, September
10, 2007)
In his critique of the RCT design, Scriven reiterates the point that
much is at stake, claiming, ‘‘This issue is not a mere academic
dispute, and should be treated as one involving the welfare of very
many people, not just the egos of a few’’ (2008, p. 24).
With so much at stake, the evidence-based movement has
fomented significant debate in recent years. The positions
espoused by participants in those debates tend to fall into two
general categories, suggesting that how questions about evidence-
based education are posed is at least as important as how they are
answered: Some discussions treat the problem primarily on a
technical-rationalistic1 level, focused on improving the fidelity of
implementation of evidence-based interventions (e.g., Galbraith
et al., 2009; Wandersman et al., 2008); others foreground the
normative and axiological nature of the problem, offering
theoretical critiques of the assumptions that undergird the very
notion of the research-practice gap (e.g., Biesta, 2007; St Pierre,
2002). Each of these approaches to posing and answering questions
about how to make education more evidence-based is helpful, yet
each is also limiting. The first leaves too much unproblematized
and risks reifying hegemonic relations of knowledge and power in
society; the second lacks grounding in practical contexts and risks
dissolving into polemical verbalism. The theoretical critiques
characterizing this second approach are necessary and helpful, but
must be supplemented by empirical studies rooted in the
particular work processes of individuals and organizations.
1.1. Purpose and research questions
To that end, the purpose of the study presented in this paper
was to better understand what actually happens, in practice, when
people try to make non-formal education more ‘‘evidence-based.’’
Like Timmermans and Berg (2003) in their analysis of standardi-
zation in medical practice, instead of debating the advantages and
1
‘‘Technical-rationality’’ is defined as a positivist epistemology of practice
whereby ‘‘professional activity consists in instrumental problem solving made
rigorous by the application of scientific theory and technique’’; it is ‘‘the view of
professional knowledge which has most powerfully shaped both our thinking about
the professions and the institutional relations of research, education, and practice’’
(Schön, 1983, p. 21). See also Usher et al. (1997).
disadvantages of evidence-based approaches and getting stuck on
a rhetorical level of analysis, I offer a study of the politics of
evidence in practice. By concentrating on the details of practice in
three cases in which people’s work involves evidence-based
programs and practices, I elucidate some of the tensions and gaps
inherent in that work, calling the apparently self-evident
superiority of evidence-based education into question. Specifically,
I address the following three research questions: (1) How is
evidence-based program and evidence-based practice work
actually practiced? (2) What perspectives and assumptions about
what non-formal education is are manifested through that work?
and (3) What conflicts emerge through that work related to those
perspectives and assumptions?
In the remainder of this paper, I first provide a brief background
discussion on the ‘‘evidence-based’’ movement, focusing especially
on the critical debates that have accompanied it. Then, I introduce
this paper’s three focal cases and present the methodological
approach I employed to explicate their politics of evidence. Third,
informed by theoretical approaches that explore the intersection of
science, expertise, and democracy, I present data and interpreta-
tions regarding the practices of the ‘‘evidence-based’’ education
movement. I focus specifically on some of the divergent and
conflicting perspectives and assumptions about what non-formal
education is that are brought to light through my data and
interpretations. In doing so, my goal is to explore the ways in which
the everyday work, the social relations, the textual mediations that
make up evidence-based educational programs and practices are
inherently and inevitably political in that—through their config-
urations of knowledge and power—they are performative and
productive; they make educational realities (the program, the
educator, the learner, etc.) to be this rather than that way.
2. Background: decades of ‘‘evidence-based’’ debates
2.1. Debates in evidence-based medicine
The increased push to use only programs and practices ‘‘that
work’’ (meaning those for which evidence of effectiveness has been
shown by way of at least one RCT) has catalyzed the proliferation of
evidence clearinghouses, which establish criteria for what counts
as credible evidence and then rank programs or other interventions
accordingly. The approach originated in biomedicine as part of the
evidence-based medicine (EBM) movement. One leading group in
that field is the Cochrane Collaboration (www.cochrane.org),
which curates a database containing systematic reviews and meta-
analyses of published RCT studies on a wide variety of health and
medical topics. EBM is ‘‘the conscientious, explicit, and judicious
use of current best evidence in making decisions about the care of
individual patients, . . . [integrating] individual clinical expertise
with the best available external clinical evidence from systematic
research’’ (Sackett, Rosenberg, Muir, & Brian, 1996, p. 71). Although
EBM is strongly linked to the RCT design, Sackett and colleagues
have admitted the need for methodological plurality: ‘‘Evidence
based medicine is not restricted to randomised trials and meta-
analyses. It involves tracking down the best external evidence with
which to answer our clinical questions’’ (Sackett et al., 1996, p. 71).
Regardless of Sackett et al.’s intentions, EBM quickly became
synonymous with RCTs and, simultaneously, became a polemic in
the health sciences. Critics railed against it as a needless neologism,
an act of intellectual poverty, a governmental management
strategy to cut health care costs, and a scientistic oversimplifica-
tion of professional health practice (Couto, 1998; Shahar, 1998).
Highlighting the fact that many other (non-randomized) sources of
evidence are obviously important in health care, a facetious article
appeared in the British Medical Journal entitled ‘‘Parachute use to
prevent death and major trauma related to gravitational challenge:
 T. Archibald / Evaluation and Program Planning 48 (2015) 137–148
Systematic review of randomised controlled trials’’ (Smith & Pell,
2003). One of EBM’s most scathing critiques comes from Holmes,
Murray, Perron, and Rail (2006), who presented the EBM
movement as ‘‘outrageously exclusionary and dangerously nor-
mative with regards to scientific knowledge . . . a good example of
microfascism at play in the contemporary scientific arena’’ (p. 181).
They described EBM as a ‘‘regime of truth’’ (as defined by Foucault)
which produces subjugated knowledge and ‘‘does not allow
pluralism, unless that pluralism is engineered by the Cochrane
hierarchy itself’’ (p. 181).
A salient and troubling aspect of the responses to Holmes
et al.’s postmodern deconstructions of EBM’s philosophical
foundations is the way in which such responses ignored the
substantive thrust of the critical arguments, dwelling instead on
the difficult to understand nature of the text (Goldacre, 2006).
Ironically justifying claims about the exclusionary discourses of
EBM, ‘‘Goldacre readily admits to having formed his assessment
of their article on the basis of ‘looking at the title’; he ‘just knows’
he can dismiss them because they disagree with EBM, a position
he characterizes in such platitudinous terms that anyone
opposing it must have misunderstood’’ (Loughlin, 2007, p.
517). The trope of ‘‘just knowing,’’ and its underlying conceptual
significance, is an idea to which I return below.
2.2. Debates in evidence-based education
In an example of ‘‘borrowed knowledge’’ (Kellert, 2008;
Longino, 1993), the discourses and practices of the evidence-
based movement were translated from biomedicine to other
domains. The field of education features prominently among the
plethora of fields which have received or developed their own
appropriation of evidence-based practice—which include evi-
dence-based management, decision making, coaching, social
services, policing, conservation, dentistry, policy, occupational
therapy, prevention science, dermatology, gambling treatment, sex
education, criminal justice, needle exchange programs, and
prices—all part of ‘‘an evidence-based global society’’ (Donaldson,
2009, p. 4). Akin to the Cochrane Collaboration in healthcare,
groups such as the Campbell Collaboration (http://www.camp-
bellcollaboration.org/) and the What Works Clearinghouse (http://
ies.ed.gov/ncee/wwc/) of the Institute of Education Sciences were
established to rate the evidence and conduct systematic reviews
and meta-analyses on educational programs.
In the field of education (especially formal education, initially),
debates about evidence-based education and RCTs were largely
catalyzed by the passage of the No Child Left Behind (NCLB) Act of
2001 (Public Law 107–110), with its explicit discursive positioning
of ‘‘scientifically based research,’’ effectively mandating RCT
designs for the research and evaluation of federally funded
education programs (Carlson & Levin, 2005; Feuer, Towne, &
Shavelson, 2002). The passage of that act was followed by two
reports commissioned by the U.S. Department of Education,
written by the National Research Council of the National Academy
of Sciences (National Research Council [NRC], 2002) and by the
Coalition for Evidence-Based Policy (Coalition for Evidence-Based
Policy [CEBP], 2003), both of which were intended to guide
educational researchers toward more ‘‘scientific’’ work. Feuer et al.
(2002), three of the NRC report’s authors, described the thinking
represented in the report alongside a handful of invited responses.
Among those responses, some authors agreed generally with Feuer
et al., yet took issue with the NRC committee’s apparently
uncritical acceptance of the charge ‘‘to define the scientific in
educational research’’ which led the committee to produce ‘‘a
statement that risks being read as endorsing both the possibility
and the desirability of taking an evidence-based social engineering
approach to educational improvement nationwide’’ (Erickson &
139
Gutierrez, 2002, p. 21). Still others took a decidedly more critical
approach to their response, seeing the NRC report as narrowly
defining ‘‘science as positivism and methodology as quantitative’’
(St Pierre, 2002, p. 25). Like Holmes et al. (2006), St. Pierre’s
postmodern analysis illustrates ‘‘the danger of the report’s
normalizing and totalizing discourse’’ (St Pierre, 2002, p. 25).
Again evoking Holmes et al., she remarked that ‘‘it is often the case
that those who work within one theoretical framework find others
unintelligible’’ (St Pierre, 2002, p. 25).
The CEBP report, unlike the NRC report, is more straightforward
in its advocacy of RCTs: ‘‘Well-designed and implemented
randomized controlled trials are considered the ‘gold standard’
for evaluating an intervention’s effectiveness, in fields such as
medicine, welfare and employment policy, and psychology. . . .
Such trials should play a similar role in education’’ (CEBP, 2003, p.
1). This discourse makes explicit the envy that the educational
researchers involved display for (their perception of) medical
science:
In the field of medicine, public policies based on scientifically-
rigorous evidence have produced extraordinary advances in
health over the past 50 years. By contrast, in most areas of social
policy . . . government programs often are implemented with
little regard to evidence, costing billions of dollars yet failing to
address critical social problems. (CEBP, 2012)
Whether and how social scientific endeavors should—or could—
answer the CEBP’s call to emulate the evidence-based medical
model, which itself is not without detractors, are questions of
central importance.
Helpful insights regarding those questions are offered in
Donaldson, Christie, and Mark (2009), What Counts as Credible
Evidence in Applied Research and Evaluation Practice. The text is
presented in two parts, one containing chapters from authors who
favor experimental approaches as the route to credible evidence
and the other with contributions from authors who favor
‘‘nonexperimental’’ approaches.2 For instance, Henry presented
his rationale for supporting use of RCTs when ‘‘getting it right
matters;’’ claiming that ‘‘the methods-based approach is deeply
rooted in the theory of representative democracy and the value
that evaluation can bring to society’’ (2009, p. 33). In the second
half of the book, Rallis (2009) considered the ethical premises for
credible evidence, adding the criterion of probity (integrity and
uprightness) to balance criteria of rigor as they pertain to
reasoning. Similarly, Greene (2009) premised her chapter ‘‘on
the understanding that evaluation is both influenced by the
political, organizational, and sociocultural context in which it is
conducted and, in turn, serves to reshape and reconstitute that
context in particular ways’’ (p. 153).
RCTs are one practice through which evaluation and research
reconstitute contexts (and other realities). The strengths, limita-
tions, and philosophical groundings of RCTs in social research have
been explicated on many occasions (Donaldson et al., 2009;
Morrison, 2009; Scriven, 2008; Walters, Lareau, & Ranis, 2009).
Scriven (2008) provided a summative evaluation of the RCT
methodology. He pointed out the ‘‘effort at persuasive redefini-
tion’’ of RCTs as ‘‘true experiments’’ and related that discursive
strategy to ‘‘the urge to create a new specialty, an esoteric cabal in
which the originators have a privileged position as keeper of the
keys to knowledge’’ (p. 12). Among his summative propositions,
Scriven first claimed that true RCTs have ‘‘essentially zero practical
application to the field of human affairs’’ (2008, p. 12), especially
because it is almost impossible to have a truly single-blind
2
The definition of the latter half of the book by way of negation (grouping
approaches by what they are not rather than what they are) problematically
‘‘others’’ those approaches, and indexes the difficulty of engaging equally with the
plurality of approaches discussed.
140 T. Archibald / Evaluation and Program Planning 48 (2015) 137–148
(let alone double-blind) study in social research; RCTs in education
are ‘‘zero-blind,’’ and thus should be called ‘‘pseudo-RCTs.’’ It is
worth noting that a few years earlier, Scriven, in his Claremont
debate with Lipsey, expressed a considerably more moderate view
of the RCT’s utility: ‘‘This is a very powerful tool, and sometimes
much the best tool, but it has as the same value as the torque
wrench in a good mechanic’s toolbox. For certain tasks, you can’t
beat it’’ (Donaldson & Christie, 2007, p. 75). But no mechanic, and
no governing body of professional mechanics, would ever deem the
torque wrench the ‘‘gold standard’’ of tools.
Scriven (2008) pointed out that ‘‘the real ‘gold standard’ for
causal claims is the same ultimate standard as for all scientific
claims; it is critical observation’’ (p. 18). This perspective is closely
aligned with the widely held belief within the field of evaluation
that the true gold standard is methodological appropriateness.
Interestingly—as Scriven discussed in a post to the EvalTalk listserv
of the American Evaluation Association (Scriven, 2013, May 29)—
Cook, Shadish, and other leading theorists behind the push to
establish the primacy of the RCT have recently explored non-
randomized designs capable of supporting causal claims (Pohl,
Steiner, Eisermann, Soellner, & Cook, 2009; Shadish, 2011; Shadish,
Rindskopf, & Hedges, 2008). Yet despite such nuanced methodo-
logical debates that occur in some quarters of academia, the ‘‘RCT
as gold standard’’ perspective still persists in a number of practical
policy settings. According to Schwandt, in those settings—as a
result of the evidence-based movement—policy discussions on
methodology are driven by ‘‘a new epistemological politics in
which some knowledge and some knowers are privileged over
others in making decisions about practices and policy . . . not [by]
the relative merits of experimental and quasi-experimental
designs’’ (Greene, Lipsey, Schwandt, Smith, & Tharp, 2007, p.
116). It is from those settings that I drew the example cases for the
study presented in this paper.
2.3. Assumptions at the intersection of science, expertise, and
democracy
Before presenting this study’s three focal cases and the
qualitative methods I used to elucidate their politics of evidence,
I first introduce some theoretical perspectives that helped frame
my approach to data collection, analysis, and interpretation. As my
review of the ‘‘evidence-based’’ debates presented above suggests,
the work to make non-formal education more ‘‘evidence-based’’ or
‘‘scientific’’ contains and relies on a number of philosophical
assumptions—on the nature of knowledge, of practice, and even of
reality—which lie at the intersection of science, expertise, and
democracy. To address my research questions, to identify and
understand the divergent perspectives on non-formal education
that are manifest through the ‘‘evidence-based’’ movement,
requires theoretical lenses that focus on uncovering and proble-
matizing tacit philosophical assumptions.
One such lens is offered by Biesta (2010), who explicated the
epistemological, ontological, and praxeological assumptions that
inhere evidence-based education. All three types of assumptions
are evident in the data on this study’s three focal cases presented
below. In the epistemological domain, evidence-based education
apparently relies on a representational epistemology, ‘‘in which
true knowledge is seen as an accurate representation of how
‘things’ are in ‘the world’’’ (Biesta, 2010, p. 494). Paraphrasing
Dewey’s critique of ‘‘the spectator theory of knowledge,’’ Biesta
argued that a more accurate view of the nature of knowledge
(especially knowledge generated through experimentation) is
offered by transactional epistemology, in which ‘‘knowledge
is not a depiction of a static world ‘out there’ . . .. It rather is
knowledge about the world in function of our interventions’’
(2010, p. 495, emphasis in the original). On this problem (or the
version of it in physics) Heisenberg wrote, ‘‘What we observe is
not nature itself, but nature exposed to our method of
questioning’’ (quoted in Law & Urry, 2004, p. 395). In this
epistemological domain, Biesta described a ‘‘knowledge deficit’’
that relates to our actual inability to know, through experimen-
tation, that ‘‘what works’’ now, in one context, will work in the
future and in other contexts.
In the ontological domain, ‘‘talk about ‘what works’ . . . operates
on the assumption of a mechanistic ontology that is actually the
exception, not the norm in the domain of human interaction’’
(Biesta, 2010, p. 497). Social systems, such as non-formal
education, can be characterized as open, semiotic, recursive
systems. Yet when proponents of evidence-based education
express their desire to replicate the impact that RCTs had on
agriculture and medicine—like Duflo and Kremer (2003) who
stated optimistically, ‘‘Just as randomized trials for pharmaceu-
ticals revolutionized medicine in the 20th Century, randomized
evaluations have the potential to revolutionize social policy during
the 21st’’ (p. 32; see also CEBP, 2012)—they apparently overlook
the ways in which ‘‘the dynamics of education are fundamentally
different from the dynamics of, say, potato growing or chemistry’’
(Biesta, 2010, p. 497). This is what Biesta referred to as the ‘‘efficacy
deficit’’ of the evidence-based movement, ‘‘indicating that in the
social domain interventions do not generate effects in a
mechanistic or deterministic way, but through processes that . . .
are open so that the connections between intervention and effect
are non-linear’’ (Biesta, 2010, p. 497). In complex, dynamic, non-
linear social systems, efficacy is made possible through ‘‘complex-
ity reduction,’’ which ‘‘has to do with the reduction of the number
of available options for action for the elements of a system’’ (Biesta,
2010, p. 497). Complexity reduction should be seen as a social
construction, not as a naturally occurring phenomenon; and ‘‘since
any attempt to reduce the number of available options for action
for the ‘elements’ within a system is about the exertion of power,
complexity reduction should therefore be understood as a political
act’’ (Biesta, 2010, p. 498).
Finally, in the praxeological domain—which has to do with the
ways in which practice, especially professional practice, can or
should be understood in terms of the application of scientific
knowledge—Biesta built on Latour’s discussions of how techno-
science succeeds and moves by rendering the world to be more like
the laboratory from which it originated (Latour, 1983). Biesta
highlighted that
to think of the impact of modern science on society in terms of
the application of scientific knowledge—which is central to the
notion of evidence-based and evidence-informed practice—at
least misses important aspects of what makes the application of
such knowledge possible (particularly the work that is needed
to transform the outside world so that knowledge becomes
applicable) and perhaps even serves as an ideology that makes
the incorporation of practices into particular networks invisi-
ble. (Biesta, 2010, p. 499)
This ‘‘application deficit’’ points to the ways in which the
application of scientific knowledge is performative and transfor-
mative, what Leach, Scoones, and Wynne (2005) called ‘‘the
tacit provisional performance of human ontologies in the making’’
(p. 13). Just as the complexity reduction that characterizes the
ontological domain of evidence-based education is a political act,
so too is the ideological shaping of society effected through the
application of scientific knowledge in the praxeological domain.
Taken together, these views of the evidence-based education
movement raise questions about whether RCTs and evidence-
based approaches are ‘‘rooted in democratic theory,’’ as Henry
(2009, p. 39) claimed, or whether they pose ‘‘a threat to democracy
itself’’ (Biesta, 2007, p. 21). Perhaps Henry and Biesta arrived at
 T. Archibald / Evaluation and Program Planning 48 (2015) 137–148
such disparate conclusions because each was working with
different democratic theories; Henry’s analysis was based on
theories of representative democracy, while Biesta’s was based on
Deweyan theories of direct deliberative democracy. From that
latter perspective, the effects of RCTs and EBPs on democracy can
also be understood at the level of ‘‘micropolitics,’’ based on the
circulation of power and knowledge in the everyday workings of
society (Foucault, 1980; Kulynych, 1997).
Biesta concluded his analysis of the epistemological, ontolog-
ical and praxeological assumptions of the evidence-based
movement by emphasizing the role of power and normativity
and calling instead for ‘‘value-based education.’’ Given the
teleological character of education, normative questions of
purpose must be primary—‘‘because if evidence were the only
base for educational practice, educational practice would be
entirely without direction’’ (Biesta, 2010, p. 500). On this topic,
Putnam (quoted in Bernstein, 1983) wrote, ‘‘A view of knowledge
that acknowledges that the sphere of knowledge is wider than
the sphere of ‘science’ seems to me to be a cultural necessity if we
are to arrive at a sane and human view of ourselves or of science’’
(p. 1). This relationship between scientific knowledge, values,
and praxis is encapsulated in Aristotle’s classification of three
types of knowledge: episteme, techne, and phronesis. As described
by Flyvbjerg (2001), episteme ‘‘concerns universals and the
production of knowledge which is invariable in time and space’’
(p. 55), techne is ‘‘craft and art, and as an activity it is concrete,
variable, and context-dependent’’ (p. 56), and phronesis is
‘‘prudence’’ or ‘‘practical common sense’’ and involves ethics
and ‘‘deliberation about values with reference to praxis. [It is]
pragmatic, variable, context-dependent’’ (p. 57). Equipped with
Biesta’s theoretical perspectives, and sensitive to the issues of
power and normativity that inevitably lie at the intersection of
science, expertise, and democracy, I turn now to the empirical
analysis of three cases in which people’s work involves efforts to
make non-formal education to be more ‘‘evidence-based.’’
3. Methods
The methodology of the study presented in this paper is
informed by the approach taken by Timmermans and Berg (2003)
in their book, The Gold Standard: The Challenge of Evidence-based
Medicine and Standardization in Health Care: ‘‘Instead of debating
the advantages and disadvantages of standardization and getting
stuck on a rhetorical level of analysis, [they] propose a study of
the politics of standardization in practice’’ (2003, p. 21). They
drew from the interactionist sociology of work, science studies,
and ethnomethodology, to examine ‘‘how evidence-based
medicine changes the practice of medicine on both a micro
and a macro level’’ (Timmermans & Berg, 2003, p. 21). Below, I
first present this study’s three focal cases and then describe the
qualitative methods used to analyze the politics of evidence at
play in each of those cases. In an effort to protect the
confidentiality of the participants of this study, I refrain from
referring to the projects, organizations, and programs by name.
3.1. Sample: three focal cases
The study presented in this paper focused on three cases, all of
which involve non-formal youth development education. Two of
the cases are projects focused specifically on adolescent sexual
health, while the third involves a more broadly focused positive
youth development organization. Of the two adolescent sexual
health projects, one takes place in the United States (Case A), while
the other takes place in Kenya (Case B). The project staff from both
Case A and Case B support the implementation of evidence-based
programs (EBPs)—well-defined, tightly scripted curricula that have
141
been evaluated with at least one RCT—designed to prevent teenage
pregnancy and sexually transmitted diseases such as HIV. They do
not implement the EBPs themselves. Rather, both projects are
positioned between a funder and the community-based educa-
tional organizations that implement specific EBPs supported (or
mandated) by that funder. Both projects offer training and
technical assistance, provide guidance on evaluation, seek to
improve the fidelity of implementation of programs, and address
other needs expressed by both the funders and the community-
based educators. The third, more general organization (Case C)
takes place in the United States. All three cases have some relation
to a university. Cases A and C are both based in a university center
focused on translational research in the social and behavioral
sciences. As such, they are very much at the interface of research
and practice. Case C is also part of the Cooperative Extension
System (CES)—in the United States, there is a system of ‘‘land
grant’’ colleges and universities established 150 years ago that for
over 100 years have interacted with the communities of their
individual states via the CES (i.e., disseminating research-based
knowledge, engaging with local communities to identify and solve
local issues, etc.).3 Case B is a project that is a partnership of a U.S.
university and a Kenyan faith-based organization.
I selected these three cases for inclusion in this study because
they are data rich—at the time the study was conducted, each of
them was experiencing increased pressure from various stake-
holders to be more ‘‘evidence-based.’’ In other words, my
approach to sampling (at the level of organizations, projects,
programs, and people) was purposive, directed at cases in which
this study’s phenomena of interest were particularly salient. More
detail on each of the three focal cases is included in the
presentation of data in Section 4. First, I describe my approach
to data collection and analysis.
3.2. Data collection
I collected qualitative data via semi-structured interviews,
observation, and document analysis. I interviewed thirty individ-
uals (roughly ten per case). Interviews lasted on average 1 h. I
digitally recorded each interview and later transcribed them
verbatim. Interview questions were designed to elicit both people’s
stories of their work practices and people’s conceptualizations and
perceptions of key constructs such as evidence, the research-
practice gap, and non-formal education. The coming together of
people’s stories of their work and people’s more abstract
conceptualizations of their work ideally opens up space for
tensions and gaps to emerge, highlighting moments and places
where seemingly universal constructs are actually enacted in a
multitude of disparate and even contradictory ways.
I observed over 100 h of discrete events such as meetings and
public lectures, plus many additional hours of everyday work
activities. Observation of the Kenyan project was limited due to
feasibility constraints, though I did observe a week-long meeting in
Nairobi that focused explicitly on the need for the project’s work to
be more ‘‘evidence-based,’’ which was a very data-rich meeting. To
record observational data, I took notes by hand and later
transcribed them or scanned the hand-written page.
In addition to interviews and observation, I also collected
textual data such as a series of memos about evidence-based
practice and EBPs written by an administrator of one of the
projects; press releases and newspaper articles; research briefs
and outreach materials; project websites; funding proposals and
requests for funding related to the projects; textual tools
3
For a review of the multiple and changing missions of the CES and the land grant
colleges and universities, including a review of the conflicting perspectives and
narratives on what CES is, see Peters (2006, 2007) and Peters et al. (2008).
142 T. Archibald / Evaluation and Program Planning 48 (2015) 137–148
(e.g., surveys, implementation checklists, communication vehicles)
that were central to the daily functioning of project staff, and more.
Language and other aspects of discourse are extremely important
given the focus of this study (Wilson, 2009). As such, my inclusion
of textual data offered an additional level of insights and
perspectives about the phenomena of interest.
3.3. Analysis
The analysis of all three types of data was supported using a
computer assisted qualitative data software program (ATLAS.ti
7.0). I coded the data using a blend of a priori codes (based on my
research questions) and ‘‘emergent’’ codes. Methodologically, I
was guided by institutional ethnography (Smith, 2005, 2006) and
by science and technology studies (STS; e.g., Mol & Berg, 1998;
Timmermans & Berg, 2003). From STS, I drew guidance from
approaches to data collection and analysis that realize the need
for empirical inquiry aimed at determining ‘‘the processes by
which certain forms of knowledge achieve a moral legitimacy and
appear to be part of the natural order’’ (Lindenbaum & Lock, 1993,
p. xiii). For instance, Shapin and Schaffer (1985) examined the
‘‘historical circumstances in which experiment as a systematic
means of generating natural knowledge arose, in which experi-
mental practices became institutionalized, and in which experi-
mentally produced matters of fact were made into the
foundations of what counted as proper scientific knowledge’’
(p. 3). They treated truth, adequacy, and objectivity as ‘‘accom-
plishments, as historical products, as actors’ judgments and
categories’’ that are ‘‘topics for our inquiry, not resources
unreflectively to be used in that inquiry’’ (p. 14). In similar
fashion, instead of departing from pre-established truths or
norms, STS scholars ‘‘try to dissect [them] from the inside, and by
empirical means. At that point, both truth and norms start to look
like parts of ongoing practices, mobilized in specific situations by
some participants and not others’’ (Mol & Berg, 1998, p. 6). How
RCTs and EBPs come to be, through various practices and
discourses, the dominant truth and norm for legitimate knowl-
edge about non-formal education is of central importance to this
paper. These methodological and theoretical perspectives guided
my approach to explicating the epistemological politics at play in
the three focal non-formal education cases presented below.
4. Results and discussion
In this section, I first present each of the three focal cases in
greater detail, providing evidence of the tensions, negotiations, and
divergent assumptions and perspectives involved in the work of
making each of them to be more ‘‘evidence-based.’’ Then, I
synthesize and interpret that evidence to draw conclusions
regarding the ways in which those conflicting perspectives and
assumptions affect non-formal educational praxis.
4.1. Case A
The project that comprises Case A originated in 2000 with
federal funding from the Department of Health and Human
Services that is administered at the state level by the state
Department of Health. The project is active across the entire state
in which it is based. According to project documents, the project
‘‘promotes positive youth outcomes and aims to reduce risky
sexual behavior among youth by advancing the principles of
positive youth development and supporting the implementation
and evaluation of evidence-based programming.’’ The project
‘‘serves as an intermediary organization that provides technical
assistance, training, and evaluation for comprehensive adolescent
pregnancy prevention programs’’ across the state. ‘‘In this capacity
project staff advance research-based practices, while also learning
directly from practitioners about the realities of implementation in
the field.’’ In the project’s early years, it ‘‘worked with community
partnerships doing youth development work with community
partnerships, broadly trying to change how communities per-
ceived and dealt with teenagers.’’ During those years, the project
was ‘‘trying to change environments, trying to change adult
attitudes, trying to sort of saturate the community with
opportunities and supports and services; and trying to really shift
this approach to a PYD [positive youth development] framework.’’
Between 2008 and 2010, the federal and state funding agencies
associated with Case A dramatically shifted their funding
priorities, triggering significant changes to the project’s work.
The funding agencies defunded the community PYD partnerships
and began funding the Case A project as an intermediary support
agency for community ‘‘providers,’’ who themselves were funded
to implement adolescent sexual health EBPs in their respective
communities. The Case A project then began to work with
58 providers around the state (e.g., community organizations,
Planned Parenthood offices, etc.) who could choose from a list of
28 EBPs. The list of EBPs was established following a review of the
evidence conducted by Mathematica Policy Research and Child
Trends, two research firms that were contracted by the federal
government. All programs on the U.S. Department of Health and
Human Services (HHS) approved list have been shown to have
evidence of positive outcomes through experimental or quasi-
experimental research designs. According to a report authored by
those two firms, the review ‘‘identifies, assesses, and rates the
rigor of program impact studies and describes the strength of
evidence supporting different program models’’ (Mathematica
Policy Research and Child Trends, 2010, p. 1). In the review, ‘‘The
highest study quality rating is reserved for randomized controlled
trials (RCTs) and similar studies that randomly assigned subjects
to the study’s research groups,’’ whereas ‘‘[q]uasi-experimental
designs with an external comparison group are eligible for at
best a moderate rating’’ (Mathematica Policy Research and Child
Trends, 2010, p. 4).
Generally, EBPs on the list are curricula that have been
developed and tested by university-based researchers, referred
to as ‘‘developers.’’ Many of the curricula were originally developed
in the late 1990s yet have been updated in subsequent years. After
the curricula are developed and proven to be effective, they are
then published and packaged by ‘‘purveyors,’’ who sell them to
community-based educators along with accompanying services,
such as trainings on the curricula. The chain of development,
testing, and production of these curricula is often referred to as ‘‘an
industry.’’ Relatedly, many people call the EBPs ‘‘education in a
box,’’ referring figuratively to the fact that the curricula are highly
scripted, formalized lesson plans, and literally to the fact that they
are sometimes sold in cardboard boxes (as a kit containing the
facilitator’s guide, a DVD, materials for youth participants, etc.). As
an intermediary organization, Case A project staff offer training,
technical assistance, and program evaluation support to the
providers.
Adaptation is a crucially important issue in the world of EBPs
(Durlak & DuPre, 2008; Firpo-Triplet & Fuller, 2012; Wandersman
et al., 2008). Case A staff must address what they call ‘‘fidelity
tension.’’ Program developers and prevention researchers are
concerned that changes in implementation of EBPs will dilute
effectiveness, while community leaders and practitioners
are concerned that ‘‘one size does not fit all.’’ Case A staff are
charged with surveilling the providers in an effort to reduce
‘‘unapproved’’ adaptations of the EBPs as they are implemented in
the field. The evaluation packets which the providers complete and
return to the Case A project include an implementation checklist in
which the providers ideally report any unapproved adaptations
 T. Archibald / Evaluation and Program Planning 48 (2015) 137–148
they have made to the curriculum. Often, the providers will report
that the EBP was implemented with perfect fidelity, yet then
informally disclose that the contingencies involved in implement-
ing the curricula in community contexts required significant
deviation from the script. Some informal unapproved adaptations
appear to be made unreflectively by community educators (e.g.,
simply omitting some activities to make the module fit into the
time allocation of school health classes, or responding to difficult
questions about sex with incorrect information), while others are
intentional, based on the expertise and agency that is expected of
skilled, professional educators (e.g., an educator who works with
gang members knows that the activities in the EBP will not retain
the attention of participants, so changes are made to render the
curriculum more accessible to the learners). In either case, staff
from the Case A project are engaged with the providers to manage
adaptations and increase the quality of implementation.
However, when one popular program on the list of approved
programs was found to focus on prevention of sexually transmitted
infections, and not on pregnancy prevention (which is the purpose
of the initiative), the program developer apparently ‘‘tacked on’’ a
curricular unit on pregnancy prevention to the existing EBP and
‘‘sent it back out.’’ But, since the original EBP was included on the
official, sanctioned list because it had undergone a ‘‘rigorous’’
evaluation (that is, with an RCT design), it should be—but
apparently was not—reevaluated after such a significant change;
this irony was not lost on project staff and some of the educators
with whom they partner. When asked why educators cannot adapt
EBPs while developers can, one study participant replied
sarcastically, the developers ‘‘just know.’’ This instance provides
a stark example of the ways knowledge hierarchies are established
and reinforced through EBP work. It echoes Goldacre ‘‘just
knowing’’ that Holmes et al.’s postmodern critique of EBM is
nonsense.
This instance reveals the complex, contingent nature of EBP
work. Behind the veneer of ‘‘science-based education,’’ it appears
that who can and cannot adapt an EBP is not about evidence
derived from an RCT, but rather is about who is in a privileged
position as the expert ‘‘knower.’’ In a similar instance that calls the
‘‘scientific’’ veneer of the EBP enterprise into question, uncertainty
exists regarding what degree of youth attendance in EBPs sessions
is sufficient to constitute participation. It is a question about
‘‘dosage.’’ In many cases, in the controlled context in which the EBP
was studied via an RCT, youth attendance was 100%. Often, this
was because the youth were given financial incentive to participate
as part of the study. Obviously, the contexts in which the EBPs are
implemented in actual community education settings differ a great
deal from those of the original study. One difference is that youth
participation is much lower. People from Case A realized that this
was an issue and sought to find out what percentage of
participation would count as a sufficient dosage. Eventually,
administrators representing one of the funding agencies involved
relayed to Case A that 75% attendance constituted sufficient
dosage. However, it is unclear what rigorous trials were conducted
to arrive at the 75% standard.
4.2. Case B
The partnership that comprises the Kenyan project started
unofficially in December, 2004 and officially in February, 2006. The
Kenyan faith-based organization had been providing a variety of
educational programs and services across Kenya for many years. In
the early 2000s, they developed a series of five modules, or lesson
plans, for HIV/AIDS prevention education. They requested funding
from the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR).
The Centers for Disease Control and Prevention (CDC), through
their office in Kenya, is responsible for implementing PEPFAR in
143
Kenya. The agency overseeing the funds that support the Case B
project is the Global HIV/AIDS Program of the U.S. Health Resources
and Services Administration (HRSA), part of HHS (the same agency
funding Case A). CDC Kenya oversees PEPFAR funding in Kenya
because HRSA does not have a presence in that country.
When the Kenyan faith-based organization contacted CDC
Kenya to receive funding, CDC then approached a research group at
a U.S. university to see if they would partner with the Kenyan
organization—such a partnership was required in order for the
Kenyan group to receive PEPFAR funding. The U.S. research group,
which was looking for potential partners for international HIV/
AIDS prevention education work, involves a small group of faculty,
staff, and graduate students. The partnership that comprises Case B
is focused on promoting youth abstinence and behavior change.
The main curriculum implemented by the Case B project is a faith-
based abstinence and behavior change curriculum targeting youth
ages 11–14. The program is extra-curricular; teachers volunteer to
participate in trainings and then implement the program with
their school children (e.g., before school, after school, on Saturdays,
etc.). Early in the partnership, the U.S.-based team worked closely
with the Kenyan team to refine and develop the existing
curriculum. In subsequent years, the U.S. team has helped to
design and deliver training, capacity building, and technical
assistance to national teams of teachers and administrators.
The curriculum, which is referred to as ‘‘homegrown’’ because it
was developed by the educators of Case B instead of emerging from
a research study, includes various characteristics of evidence-
based interventions and is rooted in health behavior change
theories, including the Social Cognitive Theory and the Theory of
Reasoned Action. Furthermore, according to Case B documents,
‘‘the learning modules and accompanying resources incorporate
Christian and traditional Kenyan and African values to more
effectively reinforce and sustain the messages.’’ In the early years,
the individuals involved in the Case B project had a fair amount of
control over how they developed and evaluated their homegrown
curriculum. But starting in the years between 2008 and 2010, the
CDC in Kenya and elsewhere in Africa increased the emphasis on
implementing evidence-based (or, ‘‘evidence-informed’’) inter-
ventions. Case B project staff became acutely aware that they had
not generated sufficiently ‘‘rigorous’’ evidence (meaning evidence
derived from an RCT) of their program’s effectiveness.
One of the initiatives undertaken by the Case B project since
2011 involves a ‘‘targeted evaluation,’’ using a pre/post design to
assess change over time on some participant self-reported
behavior outcomes with a stratified random sample of schools
across Kenya. The importance of this targeted evaluation cannot be
overstated. The evaluation is explicitly linked to the homegrown
curriculum’s continued existence. CDC, in the United States and
abroad, has a number of initiatives related to their Diffusion of
Effective Behavioral Interventions (DEBI) project, which began in
1999 after they published the first CDC compendium of evidence-
based behavioral interventions, listing programs that have
sufficient evidence of effectiveness based on ‘‘rigorous’’ evaluation
and research (with experimental or quasi-experimental designs).
Many CDC scientists and administrators related to HIV/AIDS EBP
efforts in Africa have published on this topic, focusing especially on
the adaptation of U.S.-based EBPs to African contexts (e.g.,
Fitzgerald et al., 1999; McKleroy et al., 2006; Parker et al., 2012;
Poulsen et al., 2010).
Starting in 2008, the CDC and the Kenya National AIDS and STI
Control Program (NASCOP) Evidence-Informed Behavioral Inter-
vention Technical Working Group began addressing the need to
ensure that HIV/AIDS education across the country was evidence-
based. Together they developed the Kenya HIV Prevention
Intervention Assessment Tool (KHPIAT), a document designed to
‘‘guide the systematic assessment of homegrown HIV prevention
144 T. Archibald / Evaluation and Program Planning 48 (2015) 137–148
behavioral interventions that have not been rigorously evaluated
. . . [and help] ensure that behavioral interventions implemented in
the country are of high quality and include characteristics of
effective interventions.’’ At the time this study was conducted, the
Case B project was awaiting the release of the final draft of the
KHPIAT, hoping that their curriculum would be rated as having
enough characteristics of an evidence-based intervention to
maintain its funding. Simultaneously, the Case B project was
working to complete their targeted outcome evaluation, which
they hoped would further bolster the evidence-base for their
program. According to one administrator with the CDC:
At different times, when we were really making a movement
away from things without evidence to EBIs [evidence-based
interventions], we considered at different times telling
[the project] they need to move towards one of the EBIs, but
because they have this targeted evaluation in place and because
they’ve based it on characteristics of effective programs and
they kept such a strong model with the religious tone, too,
which is so important here and culturally relevant in Kenya,
we’ve continued to let it go and try to improve the quality.
At the time my study concluded, the Case B project was still
permitted to implement their homegrown program, yet they also
were required to implement an EBP that had been adapted from a
U.S.-based program, and were still involved in collecting and
analyzing outcome data from their own randomized trial (which
was referred to sarcastically by one Case B staff person as ‘‘our
great positivist model’’). Both of these two additional sets of
activities were very taxing to the limited capacity of the project
staff, yet they were pleased that they could continue to implement
their homegrown curriculum. Additionally, they appreciated that
the quality of its implementation was increasing as a result of the
heightened emphasis on evidence-based interventions.
As with Case A, adaptation is an important issue facing EBP
work in Kenya. As mentioned above, the CDC is involved in
adapting U.S.-derived EBPs for implementation in Kenya, a process
they refer to as ‘‘Kenyanizing’’ the curricula. This raises two related
questions. First, there is a question of external validity. If a program
worked with African American youth in Baltimore, should it be
expected to work with African youth in Kisumu? Second, how
much adaptation is necessary and sufficient? Are there some
aspects of the curriculum—such as the philosophical assumptions
implicit in the underlying theories (e.g., Social Cognitive Theory
and the Theory of Reasoned Action)—that are essentially universal,
and others—such as the names and verbal mannerisms of the youth
in the curriculum’s examples and vignettes—that are culturally
specific? How do we know which elements of the curriculum are
universal and which require adaptation? The CDC attempts to be
systematic about the process of Kenyanization, but has experi-
enced some difficulties. For instance, a CDC administrator
described observing the implementation of a U.S.-derived EBP
that had supposedly been adapted to the Kenyan context, yet
which still contained American mannerisms, concluding that ‘‘if
you start with the Kenyan process, it is so much more Kenyan then
taking an American product and trying to make it Kenyan.’’
Adapted U.S.-derived EBPs ‘‘still don’t come across with the same
Kenyan sound as probably [the partnership’s homegrown curricu-
lum] does.’’
Another difficulty in adaptation arose when a U.S.-derived EBP
was adapted and split into two versions, one for use with primary-
school age Kenyans (which promoted abstinence and behavior
change) and another for use with older youth (which included
promotion of condom use). Community educators were trained on
both versions together, and then some educators mistakenly went
into a Catholic elementary school and implemented the version
including condoms. The backlash from the local school officials and
the parents was significant. Again, this highlights the nuances
involved in ‘‘Kenyanizing’’ imported EBPs and shows where locally
developed curricula may have an advantage.
4.3. Case C
Case C is focused on an organization that does not work with
adolescent sexual health EBPs, but rather takes a broader approach
to working with youth. As such, by comparing it to Cases A and B,
divergent and conflicting perspectives and assumptions about
what non-formal education is and about how it should be informed
by evidence become apparent. As mentioned above, the organiza-
tion comprising Case C is the youth development initiative of
one state’s Cooperative Extension system. As such, it is active
across the entire state in which it is located, and is also affiliated
with the national Extension system. In one sense, as part of the
land-grant mission, the organization has been connecting research
and practice for one hundred years. Yet especially because of
shrinking financial resources, and because in 2011 the Case C
organization became affiliated with a center for translational
research in the social and behavioral sciences, the organization was
experiencing an increased emphasis on being evidence-based. One
university-based administrator expressed an interest in only
allowing the organization’s community-based educators to imple-
ment EBPs that were connected to university faculty’s research. In
effect, this would have meant eliminating the Case C organization,
since there are practically no EBPs that fit that criterion.
The Case C organization appears to be incommensurable with
the educational framework that EBPs represent. For example, the
organization emphasizes research-based practices—e.g., the prac-
tices associated with positive youth development, or PYD (Damon
& Gregory, 2002; Delgado, 2002; Hamilton, Hamilton, & Pittman,
2004; Jelicic, Bobek, Phelps, Lerner, & Lerner, 2007; Lerner &
Benson, 2003; Pittman, Irby, & Ferber, 2000; Roth & Brooks-Gunn,
2003)—rather than research-based programs. Relatedly, while
some specific outcomes such as reduced teen pregnancy or
increased interest in science are important to the Case C
organization, it tends to focus more on providing a positive
environment for youth so as to foster broader PYD outcomes such
as a youth’s sense of belonging, mastery, independence, and
generosity. This begs the question: Are some educational
initiatives and some social phenomena intrinsically more amena-
ble to an RCT design and to being packaged as an EBP?
Educators involved in the Case C organization—like some
people involved with Cases A and B as well—question the theory of
change that is manifest in an EBP. They do not think that if a youth
attends 75% of eight 2-h modules, then that youth will make
positive changes to her or his behavior. Summarizing this point,
one administrator involved in Case A stated:
I have a healthy skepticism about behavior change to begin
with, and behavior change among young people and the actual
ultimate effectiveness of a curriculum that’s going to change
behaviors, such basic behavior as sexual activity . . . based on
some intervention provided for four weeks or six weeks. . . . I
think the impetus, especially at the federal level is they wanted
to be able to document to Congress that the money that they
were spending on this wasn’t just a total waste, so . . . they
created a whole industry of [EBPs] that comes with the training
and buying the curriculum; people are experts. . . . I know the
intention is good, I just think the execution is shoddy and the
theory behind it is tenuous at best about that whole, providing
education is going to lead to behavior change, especially long
term behavior change.
People such as this administrator and the educators involved in
Case C perceive the complex and dynamic nature of youths’ lives
 T. Archibald / Evaluation and Program Planning 48 (2015) 137–148
and try to craft educational opportunities that respond to those
realities.
Importantly, during the time this study was conducted, the
educators of Case C had the agency to create their own homegrown
programs and did not need to choose a curriculum from a list of
EBPs. According to many Case C staff, this allows them to be more
responsive to community needs. One Case C administrator,
speaking about Cooperative Extension in general, describes their
overarching approach to community education this way: (1)
people in the community identify the priorities that matter to
them; (2) they then think in terms of how they want to address
that issue—the solutions that they might come up with are theirs;
(3) they put their own resources on the table; (4) if they bring in
other resources from the outside, they are brought in on local
people’s terms; and (5) there is an institutional underpinning for
the work that gets done. It is important to note, however, that
Cooperative Extension is often also perceived more as a
‘‘technology-transfer’’ system, and in that way is seen to be more
similar to EBP-implementing organizations. It is a debate that
continues within the Cooperative Extension system.
4.4. Divergent perspectives and assumptions about non-formal
education
Applying Biesta’s theoretical analysis to the data presented
above can help me analyze the epistemological and ontological
politics at play these cases, particularly in regards to the divergent
philosophical perspectives and assumptions that underlie (and are
reproduced by) those practices. For example, data about the
funder-mandated EBPs with which the projects in Cases A and B
work suggest that the creation and dissemination of those
programs manifest and tacitly assume a representational episte-
mology. Those specific EBPs were created and evaluated in one
specific context under explicitly manipulated conditions, produc-
ing knowledge of the programs’ outcomes that is then assumed to
hold true over time and space, as long as the EBP is implemented
with fidelity. Yet the representational epistemology that is tacitly
assumed ignores the multitude of ways in which the RCTs which
earned those EBPs their ‘‘evidence-based’’ status were actually
interventions. Those experiments were highly transactional. To
think that the knowledge created about an EBP’s outcomes through
the transactional processes of an RCT can somehow be generalized
into future implementations of that same curriculum, under vastly
different contexts, requires a rather large leap of epistemic logic. It
is a problem of external validity and generalizability.
In the ontological and praxeological domains, the data indicate
that the developers, purveyors, and funders involved in implement-
ing the EBP work in Cases A and B tacitly rely on mechanistic notions
of behavior change that are at the heart of the broader ‘‘evidence-
based’’ movement. Interestingly, when participants in my study
spoke explicitly to these assumptions about the nature of human
behavior change, it was to critique the EBP endeavors in which they
were involved. For example, the staff of the project in Case A spoke
longingly about their years spent working on holistic community-
wide PYD efforts. They indicated that such work was much more in
tune with their actual espoused theories of behavior change. As
quoted previously, one administrator expressed ‘‘healthy skepti-
cism’’ of the theories that frame the entire EBP endeavor, calling
them ‘‘tenuous at best.’’ One staff member put it this way:
When you think about social behavior or community based
behavior there are so many factors that influence that. And
when you take a rigorous research method, you’re trying to
eliminate all those factors and you focus on the one or two. And
so that’s not that realistic. There’s no community, there’s no
social behavior that’s just influenced by one thing.
145
In similar fashion, both the Kenyan and American staff involved in
Case B, based on their professional expertise in non-formal
education in Kenya, perceived that the EBP model (especially
with its privileging of U.S.-derived EBPs) ignored or effaced many
cultural and contextual complexities that are fundamentally
important considerations in their praxis. There are stark, unan-
swered ontological (and ethical) questions that surround the
process of ‘‘Kenyanizing’’ an EBP. How much adaptation is
necessary and sufficient? How do we know which elements of
the curriculum are universal and which require adaptation? Some
could (and do) treat these as technical, empirical questions. Yet
values, norms, and the ontological status of culture must somehow
be admitted into such conversations as well.
Evidence of the ways in which the application deficit, in the
praxeological domain, changes educational realities through EBP
work was plentiful. For example, the project in Case A fundamen-
tally shifted its mode of working and its pedagogical theory-in-
action when it converted to the EBP model. The community
educators with whom the project staff work did so as well. The
identity and subjectivity of what it means to do adolescent sexual
health education work across the entire state took on completely
different ontological and pedagogical forms. Encapsulating this
point, one university researcher not directly associated with the
project, yet who was previously a community-based adolescent
sexual health educator, learned that the current EBP model
requires educators to literally read from the curriculum’s script
and asked (incredulously and rhetorically), ‘‘Well, who would ever
want to be an educator then?!’’ It is important to note that some
community educators like being able to ‘‘hang their hats’’ on the
EBP, yet they simultaneously realize that they have become less
able to respond to community needs and to exercise their
professional judgment. Concerning the project in Case B, the
homegrown program would have been cut if they did not engage in
the EBP movement’s model, one stark way in which the application
of scientific knowledge remakes the world. One less obvious way it
does so pertains to how the identities and subjectivities of Kenyan
youth might be remade depending on which type of sexual health
education is offered to; are the ‘‘Kenyanized’’ EBPs the project
implements overly ‘‘Americanizing’’ the Kenyan youth? Concern-
ing the organization in Case C, if the administrator who stated that
they should only implement EBPs derived from university research
had implemented that plan, the organization would have been
eliminated. Corroborating Latour’s and Biesta’s claims, these
examples show that the application of scientific knowledge in
society is ontologically performative and transformative. Reflect-
ing on these examples, I agree with Biesta that the application
deficit involves ‘‘ideology that makes the incorporation of practices
into particular networks invisible’’ (Biesta, 2010, p. 499).
In light of the analyses presented above, in order to synthesize
and succinctly articulate my comparison of these three cases, I
parsed the themes that I identified about divergent perspectives
and assumptions on what non-formal education is into two
fundamentally different modes of education: Mode 1 and Mode 2
(see Table 1). The primary axis of difference, which relates to
Biesta’s discussion of the inevitably teleological character of
education, is whether non-formal education is seen and done as an
infrastructure for the dissemination of scientific information or as a
site of grassroots knowledge sharing. This first axis is closely related
to a second one regarding different perspectives about what the
essential unit of educational interaction is: A program (meaning a
tightly bounded and scripted curriculum) or a set of practices. As I
present in the table, these two axes of difference relate to other
variable ways of seeing and doing non-formal education.
It is important to note that, in practice, the two modes are
not (usually) so starkly dichotomized as this table suggests. In the
day-to-day contingencies of people’s work processes, the two tend
146 T. Archibald / Evaluation and Program Planning 48 (2015) 137–148
Table 1
Divergent perspectives and assumptions about what non-formal education is.
Mode 1
Non-formal education is. . . An infrastructure for the dissemination of scientific information
Program planning and evaluation decisions are. . . Campus- or scientist-driven
The essential unit of educational interaction is. . . A program (meaning a tightly bounded and scripted curriculum)
Behavior change is assumed to be. . . Simple, or complicated; linear
The focus is on. . . Content delivery and specific outcomes
to be blended together, with one or the other being more or less
emphasized depending on the particular situation. However, the
table offers a simplified heuristic overview of these divergent
perspectives and assumptions that constitute a key finding of my
study.
Both modes value evidence and knowledge. Both are motivated
by democratic and ethical concerns. Yet each mode engages with
its democratic and ethical project differently. For instance, EBPs
and Mode 1 education value episteme over techne and phronesis.
Mode 1 education seeks to advance democracy by directing and
justifying action with scientific knowledge and technique, an
example of the processes ‘‘through which science and technology
have been harnessed to the development of modern liberal-
democratic concepts of action, authority, and accountability’’
(Ezrahi, 1990, p. 1). Mode 2 education, on the other hand, values
both techne and phronesis over episteme. It engages with its
democratic project by questioning abstract hierarchies of knowl-
edge and power. Mode 2 education is concerned with cognitive, or
epistemic justice, defined as ‘‘the constitutional right of different
systems of knowledge to exist as part of dialogue and debate’’
(Visvanathan, 2005, p. 92). It ‘‘has to do with the coexistence of
many knowledges in the world and the relation between the
abstract hierarchies which constitute them and the unequal
economic and political power relations which produce and
reproduce increasingly more severe social injustice’’ (Toulmin,
2007, p. xv). According to Visvanathan, ‘‘One has to realize that
epistemology is not a remote, exotic term. It determines life
chances’’ (2005, p. 84), which evokes Scriven’s claim regarding
what is at stake in the ‘‘gold standard’’ debates: ‘‘This issue is not a
mere academic dispute, and should be treated as one involving the
welfare of very many people, not just the egos of a few’’ (2008, p.
24). As such, what Table 1 helps put into sharp relief is that, while
the conflicts and tensions between these divergent perspectives
may appear academic or irrelevant, they actually have stark
implications for what non-formal education is and can be in
society.
5. Conclusions: lessons learned
The study described here suggests that the current dominant
approach to making non-formal education more evidence-based
by way of EBPs is seriously flawed. The self-evident superiority of
evidence-based programs must be revisited. In this study, I found
that the practices and processes that comprise EBP work are far
from ‘‘scientific,’’ in the honorific sense of the term: university-
based researchers develop and test EBPs in contexts that differ
vastly from those where the EBPs are eventually implemented,
seriously stretching the philosophical limits of external validity
and generalizability; the list of approved EBPs for a pregnancy
prevention initiative includes a curriculum that makes no mention
of pregnancy; unapproved accidental adaptations as well as
unapproved purposeful adaptations are commonplace in the
settings where community educators implement the EBPs; tacitly
(and paradoxically) approved adaptations are made by the
developers, apparently based on their more privileged position
as expert knowers, while community educators feel limits placed
Mode 2
A site of grassroots knowledge sharing
Community-driven
A set of practices and processes
Complex; non-linear
Process facilitation and general outcomes
on their agency to respond and adapt to the needs and realities in
their local communities; and project administrators admit that the
EBP model’s espoused theories of behavior change, processes of
‘‘Kenyanization’’ and adaptation, and overall efficacy are all highly
questionable.
At least two ironies emerge related to my conclusions about the
contingent and messy implementation of ‘‘science-based educa-
tion.’’ One is that, because the EBPs have already been proven to be
effective (in some cases, by way of one RCT fifteen or twenty years
ago), the curricula are generally implemented in communities with
no outcome evaluation whatsoever. Another irony is that the
‘‘evidence-based education’’ movement itself is non-experimental,
since it has not undergone an RCT and has not been proven to be
better than alternative approaches; thus it ‘‘reflects the introduc-
tion of ideology into the heart of the empirical paradigm’’
(Chelimsky, 2007, p. 32; see also Scriven, 2008). If the dominant
approach to making non-formal education more evidence-based
by way of EBPs is so flawed, what alternative approaches are
available to program planners, administrators, researchers, and
evaluators?
I propose instead a refocusing of the field’s efforts on a more
methodologically pluralistic approach involving support for
evidence-based practices. Doing so will require more nuanced
conceptualizations of grades of evidence (Chatterji, 2007), varieties
of validity (Chen, 2010; Cronbach, 1982), and definitions of rigor
(Urban, Hargraves, & Trochim, 2014). This will perhaps come more
naturally to (or already be the norm for) program planners,
educators, administrators, researchers, and evaluators who are
primarily engaged in Mode 2 educational contexts. Yet there are
ways to engender methodological plurality and practice-focused
evaluation in Mode 1 contexts, too. One suggestion for how to do
this was articulated by the president of the William T. Grant
Foundation in their 2009 annual report. The foundation supports
the goal of evidence-based youth development education, but they
‘‘have come to question the predominant approach to achieving
that goal. That approach assumes that widespread improvement in
child and youth outcomes will occur through ‘scaling-up’ brand-
name programs, models, and organizations . . . and its track record
is modest at best’’ (Granger, 2010, p. 5). According to Granger:
In today’s vernacular, we need more research attention paid to
why things work as the missing ingredient in the ‘what works’
agenda. . . . Despite the research community’s ability to identify
promising programs, our Foundation has become increasingly
convinced that it is not possible to take such programs to scale
in a way that maintains their effectiveness. (2010, pp. 5–6,
emphasis in the original)
The shift in focus to why things work ‘‘led the Foundation to
examine how policies and programs shape the daily experiences
that are the pathways to individual-level change’’ (Granger, 2010,
p. 8), phenomena which are not amenable to study via an RCT.
Their new research agenda focuses on how resources, such as
‘‘human, social, economic, cultural, and physical capital, . . . seem to
condition the social interactions in a setting in a way that creates
and reinforces setting-level phenomena—such as norms and
routines—that lead to differences in youth outcomes’’ (Granger,
 T. Archibald / Evaluation and Program Planning 48 (2015) 137–148
2010, p. 8). To do so, they encourage their grantees ‘‘to integrate
qualitative and quantitative methods to better theorize the
processes and mechanisms that lead to improved youth outcomes
. . . [and] support work on some of the methodological issues
inherent in assessing the precision of these setting-level measures’’
(Granger, 2010, p. 9). In this area of program evaluation, the W. T.
Grant foundation offers concrete alternatives in response to the
flaws of the dominant EBP methods.
Another suggestion for how to respond to the short-comings of
the dominant EBP model comes from Donald Campbell. Ironically—
based on Campbell’s later work—there is reason to believe that he
might disparage the evidence-based education approaches favored
by the clearinghouse that bears his name. Campbell took issue with
approaches that employ designated operational definitions for
theoretical terms. He saw such definitional operationalism as a
‘‘dogma’’ which ‘‘discouraged support for exploring multiple
approaches to measuring ‘the same thing’ . . .’’ (Campbell, 1984,
p. 18). In other words, such approaches are epistemically unjust.
Campbell called definitional operationalism an ‘‘unmitigated
disaster,’’ imported from logical positivism, ‘‘which persists long
after the substantial revision or rejection of positivism within the
philosophy of science. It persists most perniciously in social policy
science, in the accountability movement, or in managerial control
efforts employing single explicit quantitative criteria’’ (Campbell,
1984, p. 18). While the Campbell Collaboration is not based solely
on definitional operationalism, its methodological and evidentiary
hierarchies—with their over-emphasis on RCTs—do lead to the
same sort of conceptual narrowing that Campbell warns against.
As an alternative, Campbell suggested that we need a ‘‘cross-
validational model of program dissemination and validation,’’ with
‘‘large numbers of decentralized local innovators and independent
adopters, independently making the many ad hoc decisions about
implementation and measurement’’ (1984, p. 19). We should
support ‘‘a heterogeneity of programs, each evaluating themselves
until they feel they have a package worth others borrowing, and
support those who borrow to cross-validate the efficacy’’ (Camp-
bell, 1984, p. 19). In this truly experimenting society, evaluative
thinking and reasoning would trump methodological dogmatism.
Instead of the epistemological and pedagogical monoculture
represented by the EBP movement, heeding Campbell’s call would
foster methodological and epistemological plurality and would
leverage evaluation’s democratic potential. The question remains:
How could the field of evaluation best operationalize Campbell’s
cross-validational model? Answering that question is beyond the
scope of this paper.
In conclusion, if the RCT is no longer considered the gold
standard, and if we realize that the dominant approach to making
education more evidence-based by way of EBPs is seriously flawed,
then more phenomena (especially phenomena that involve open,
recursive, dynamic, non-linear, values-explicit practices and
processes) could be legitimated as realities that are worthy to
be researched and evaluated, and thus valued as worthy to exist. I
hope that the theoretical and empirical analysis of the three cases
presented above helps open up dialog on questions about what
education is, what research is, what science is, what practice is, and
what evaluative perspectives should be brought to bear on all of
those questions. Such discussions are crucial if we are to seek a
more effective and ethical praxis of evaluation and program
planning which embraces rather than effaces the intricacies and
nuances that characterize social human action.
Acknowledgements
This study is drawn from the author’s doctoral dissertation. The
author would like to thank the members of the thesis committee
for their guidance on this project: Arthur Wilson, William Trochim,
147
and Stacey Langwick. The author is grateful to Dr. Trochim and to
four anonymous reviewers for their very helpful comments on an
earlier version of this article. The author would also like to thank all
those who participated in this study. The study was supported in
part by a Graduate Research Assistantship funded through
National Science Foundation Award # 0814364 and by the
Institute for African Development at The Mario Einaudi Center
for International Studies.
References
ATLAS.ti 7.0 (2014). ATLAS.ti scientific software development. Berlin: GmbH.
Bernstein, R. J. (1983). Beyond objectivism and relativism: Science, hermeneutics, and
praxis. Philadelphia: University of Pennsylvania Press.
Biesta, G. J. J. (2007). Why ‘‘what works’’ won’t work: Evidence-based practice and the
democratic deficit in educational research. Educational Theory, 57(1), 1–22.
Biesta, G. J. J. (2010). Why ‘‘what works’’ still won’t work: From evidence-based
education to value-based education. Studies in Philosophy and Education, 29(5),
491–503.
Campbell, D. T. (1984). Science policy from a naturalistic sociological epistemology.
PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association (pp.
14–29). Philosophy of Science Association.
Carlson, J. S., & Levin, J. R. (2005). The No Child Left Behind legislation: Educational
research and federal funding. Greenwich, CT: Information Age Publishing.
Chatterji, M. (2007). Grades of evidence: Variability in quality of findings in effective-
ness studies of complex field interventions. American Journal of Evaluation, 28(3),
239–255.
Chelimsky, E. (2007). Factors influencing the choice of methods in federal evaluation
practice. In G. Julnes, & D. Rog (Eds.), Informing federal policies on evaluation
methodology: Building the evidence base for method choice in government sponsored
evaluation. New directions for evaluation (Vol. 113, pp. 13–33). .
Chen, H. T. (2010). The bottom-up approach to integrative validity: A new perspective
for program evaluation. Evaluation and Program Planning, 33(3), 205–214.
Coalition for Evidence-based Policy (CEBP) (2003). Identifying and implementing edu-
cational practices supported by rigorous evidence: A user friendly guide. Washington,
DC: US Dept of Education Institute of Education Sciences National Center for
Education Evaluation and Regional Assistance.
Coalition for Evidence-Based Policy (CEBP) (2012). Our mission. Retrieved from http://
coalition4evidence.org/
Couto, J. S. (1998). Evidence-based medicine: A Kuhnian perspective of a transvestite
non-theory. Journal of Evaluation in Clinical Practice, 4(4), 267–275.
Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San
Francisco: Jossey-Bass Publishers.
Damon, W., & Gregory, A. (2002). Bringing in a new era in the field of youth develop-
ment. In R. Lerner, F. Jacobs, & D. Wertlieb (Eds.), Handbook of applied developmental
science (vol. 1, pp. 407–420). New York: Wiley.
Delgado, M. (2002). New frontiers for youth development in the twenty-first century. New
York: Colombia University Press.
Donaldson, S. I. (2009). In search of the blueprint for an evidence-based global society.
In S. I. Donaldson, C. A. Christie, & M. M. Mark (Eds.), What counts as credible
evidence in applied research and evaluation practice? (pp. 2–18). Los Angeles: Sage.
Donaldson, S. I., & Christie, C. A. (2007). The 2004 Claremont bebate: Lipsey vs. Scriven.
Journal of Multidisciplinary Evaluation, 2(3), 60–77.
Donaldson, S. I., Christie, C. A., & Mark, M. M. (Eds.). (2009). What counts as credible
evidence in applied research and evaluation practice? Los Angeles: Sage.
Duflo, E., & Kremer, M. (2003). Use of randomization in the evaluation of development
effectiveness. Paper prepared for the World Bank Operations Evaluation Department
(OED) conference on evaluation and development effectiveness in Washington, D.C.,
15–16 July. Retrieved from http://economics.mit.edu/files/765
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the
influence of implementation on program outcomes and the factors affecting
implementation. American Journal of Community Psychology, 41(3–4), 327–350.
Erickson, F., & Gutierrez, K. (2002). Comment: Culture, rigor, and science in educational
research. Educational Researcher, 31(8), 21–24.
Ezrahi, Y. (1990). The descent of Icarus: Science and the transformation of contemporary
democracy. Cambridge, MA: Harvard University Press.
Feuer, M. J., Towne, L., & Shavelson, R. J. (2002). Scientific culture and educational
research. Educational Researcher, 31(8), 4–14.
Firpo-Triplet, R., & Fuller, T. (2012). General adaptation guidance: A guide to adapting
evidence-based sexual health curricula. ETR Associates and the Centers for Disease
Control and Prevention Division of Reproductive Health.
Fitzgerald, A. M., Stanton, B. F., Terreri, N., Shipena, H., Li, X., Kahihuata, J., et al. (1999).
Use of Western-based HIV risk-reduction interventions targeting adolescents in an
African setting. Journal of Adolescent Health: Official Publication of the Society for
Adolescent Medicine, 25(1), 52–61.
Flyvbjerg, B. (2001). Making social science matter: Why social inquiry fails and how it can
succeed again. Oxford, UK: Cambridge University Press.
Foucault, M. (1977). Discipline and punish: The birth of the prison. New York: Pantheon
Books.
Foucault, M. (1980). Power/knowledge. Brighton: Harvester.
Galbraith, J. S., Stanton, B., Boekeloo, B., King, W., Desmond, S., Howard, D., et al. (2009).
Exploring implementation and fidelity of evidence-based behavioral interventions
148 T. Archibald / Evaluation and Program Planning 48 (2015) 137–148
for HIV prevention: Lessons learned from the Focus on Kids diffusion case study.
Health Education & Behavior, 36(3), 532–549.
Goldacre, B. (2006, August). Objectionable ‘objectives’. In The guardian Retrieved from
http://www.guardian.co.uk/science/2006/aug/19/badscience.uknews
Granger, R. C. (2010). Improving practice at scale. In William T. Grant Foundation
2009 annual report. New York, NY: William T. Grant Foundation Retrieved from
http://wtgrantfoundation.org/about_us/annual_reports
Greene, J. C. (2009). Evidence as ‘‘proof’’ and evidence as ‘‘inkling’’. In S. I. Donaldson, C.
A. Christie, & M. M. Mark (Eds.), What counts as credible evidence in applied research
and evaluation practice? (pp. 153–167). Los Angeles: Sage.
Greene, J. C., Lipsey, M. W., Schwandt, T. A., Smith, N. L., & Tharp, R. G. (2007). Method
choice: Five discussant commentaries. In G. Julnes, & D. Rog (Eds.), Informing
federal policies on evaluation methodology: Building the evidence base for method
choice in government sponsored evaluation. New directions for evaluation (Vol. 113,
pp. 111–127). .
Hamilton, S., Hamilton, M., & Pittman, K. (2004). Principles for youth development. In S.
F. Hamilton & M. A. Hamilton (Eds.), The handbook of youth development: Coming of
age in American communities (pp. 3–22). Thousand Oaks, CA: Sage.
Henry, G. T. (2009). When getting it right matters: The case for high-quality policy and
program impact evaluations. In S. I. Donaldson, C. A. Christie, & M. M. Mark (Eds.),
What counts as credible evidence in applied research and evaluation practice? (pp. 32–
50). Los Angeles: Sage.
Holmes, D., Murray, S. J., Perron, A., & Rail, G. (2006). Deconstructing the evidence-
based discourse in health sciences: Truth, power and fascism. International Journal
of Evidence-Based Healthcare, 4(3), 180–186.
Jelicic, H., Bobek, D., Phelps, E., Lerner, R., & Lerner, J. (2007). Using positive youth
development to predict contribution and risk behaviors in early adolescence:
Findings from the first two waves of the 4-H Study of Positive Youth Development.
ral Development, 31(3), 263–273.
Kellert, S. H. (2008). Borrowed knowledge: Chaos theory and the challenge of learning
across disciplines. Chicago: University of Chicago Press.
Kulynych, J. J. (1997). Performing politics: Foucault, Habermas, and postmodern
participation. Polity, 30(2), 315–346.
Latour, B. (1983). Give me a laboratory and I will raise the world. In K. D. Knorr & M.
Mulkay (Eds.), Science observed (pp. 141–170). London: Sage.
Law, J., & Urry, J. (2004). Enacting the social. Economy and Society, 33(3), 390–410.
Leach, M., Scoones, I., & Wynne, B. (2005). Introduction: Science, citizenship, and
globalization. In M. Leach, I. Scoones, & B. Wynne (Eds.), Science and citizens:
Globalization and the challenge of engagement (pp. 1–14). London: Zed Books.
Lerner, R. M., & Benson, P. I. (2003). Developmental assets and asset-building communi-
ties: Implications for research, policy, and practice. New York: Kluwer Academic/
Plenum.
Lindenbaum, S., & Lock, M. M. (1993). Knowledge, power, and practice: The anthropology
of medicine and everyday life. Berkeley: University of California Press.
Longino, H. (1993). Subject, power, and knowledge: Description and prescription in
feminist philosophies of science. In L. Alcoff & E. Potter (Eds.), Feminist epistemol-
ogies (pp. 101–120). New York: Routledge.
Loughlin, M. (2007). Style, substance, Newspeak ‘and all that’: A commentary on
Murray et al. (2007) and an open challenge to Goldacre and other ‘offended’
apologists for EBM. Journal of Evaluation in Clinical Practice, 13(4), 517–521.
Mathematica Policy Research and Child Trends (2010). Identifying programs that impact
teen pregnancy, sexually transmitted infections, and associated sexual risk behaviors.
Review protocol, Version 1.0. Retrieved from http://www.hhs.gov/ash/oah/oah-
initiatives/teen_pregnancy/db/eb-programs-review-v1.pdf
McKleroy, V. S., Galbraith, J. S., Cummings, B., Jones, P., Harshbarger, C., Collins, C., et al.
(2006). Adapting evidence-based behavioral interventions for new settings and
target populations. AIDS Education & Prevention, 18(Suppl.), 59–73.
Mol, A., & Berg, M. (1998). Differences in medicine: An introduction. In M. Berg & A. Mol
(Eds.), Differences in medicine: Unraveling practices, techniques, and bodies (pp. 1–
12). Durham, NC: Duke University Press.
Morrison, K. (2009). Causation in educational research. London: Routledge.
Mosteller, F., & Boruch, R. (Eds.). (2002). Evidence matters: Randomized trials in education
research. Brookings Institution.
National Research Council (NRC) (2002). Scientific research in education. Washington,
DC: National Academy Press.
Parker, L., Maman, S., Pettifor, A., Chalachala, J. L., Edmonds, A., Golin, C. E., et al. (2012).
Adaptation of a US evidence-based positive prevention intervention for youth
living with HIV/AIDS in Kinshasa, Democratic Republic of the Congo. Evaluation and
Program Planning, 36(1), 124–135.
Peters, S. J. (2006). ‘Every farmer should be awakened’: Liberty Hyde Bailey’s vision of
agricultural extension work. Agricultural History, 80(2), 190–219.
Peters, S. J. (2007). Changing the story about higher education’s public purposes and work:
Land-grants, liberty, and the Little Country Theater. Foreseeable futures position paper
#6, published by Imagining America: Artists and Scholars in Public Life. Retrieved from
http://imaginingamerica.org/fg-item/changing-the-story-about-higher-education
Peters, S. J., Alter, T. R., & Schwartzbach, N. (2008). Unsettling a settled discourse:
Faculty views of the meaning and significance of the land-grant mission. Journal of
Higher Education Outreach and Engagement, 12(2), 33–66.
Pittman, K., Irby, M., & Ferber, T. (2000). Unfinished business: Further reflections on a
decade of promoting youth development. In N. Jaffe (Ed.), Youth development:
Issues, challenges, and directions (pp. 17–64). Philadelphia, PA: Public/Private
Ventures.
Pohl, S., Steiner, P., Eisermann, J., Soellner, R. R., & Cook, T. D. (2009). Unbiased causal
inference from an observational study: Results of a within-study comparison.
Educational Evaluation and Policy Analysis, 31(4), 463–479.
Poulsen, M. N., Vandenhoudt, H., Wyckoff, S. C., Obong’o, C. O., Ochura, J., Njika, G., et al.
(2010). Cultural adaptation of a US evidence-based parenting intervention for rural
Western Kenya: From parents matter! To families matter! AIDS Education and
Prevention, 22(4), 273–285.
Public Law 107-110 (2014). No Child Left Behind Act of 2001.
Rallis, S. (2009). Reasoning with rigor and probity: Ethical premises for credible
evidence. In S. I. Donaldson, C. A. Christie, & M. M. Mark (Eds.), What counts as
credible evidence in applied research and evaluation practice? (pp. 168–180). Los
Angeles: Sage.
Roth, J. L., & Brooks-Gunn, J. (2003). What exactly is a youth development program?
Answers from research and practice. Applied Developmental Science, 7(2),
94–111.
Sackett, D. L., Rosenberg, W. M. C., Muir, G. J. A., & Brian, H. R. (1996). Evidence based
medicine: What it is and what it isn’t. British Medical Journal, 312(7023), 71.
Schön, D. (1983). The reflective practitioner. How professionals think in action. London:
Temple Smith.
Scriven, M. (2008). A summative evaluation of RCT methodology: An alternative
approach to causal research. Journal of Multidisciplinary Evaluation, 5(9),
11–24.
Scriven, M. (2013, May). Re: Federal evaluation policy for selected grant programs and the
definition of ‘‘evidence.’’. Email message to the EvalTalk list serve.
Shadish, W. R. (2011). Randomized controlled studies and alternative designs in
outcome studies: Challenges and opportunities. Research on Social Work Practice,
(6), 636–643.
Shadish, W. R., Rindskopf, D. M., & Hedges, L. V. (2008). The state of the science in the
meta-analysis of single-case experimental designs. Evidence-Based Communication
Assessment and Intervention, 2(3), 188–196.
Shahar, E. (1998). Evidence-based medicine: A new paradigm or the Emperor’s new
clothes? Journal of Evaluation in Clinical Practice, 4(4), 277–282.
Shapin, S., & Schaffer, S. (1985). Leviathan and the air-pump: Hobbes, Boyle, and the
experimental life. Princeton, NJ: Princeton University Press.
Smith, D. E. (2005). Institutional ethnography: A sociology for people. Walnut Creek, CA:
AltaMira Press.
Smith, D. E. (Ed.). (2006). Institutional ethnography as practice. Lanham, MD: Rowman &
Littlefield.
Smith, G. C., & Pell, J. P. (2003). Parachute use to prevent death and major trauma
related to gravitational challenge: Systematic review of randomised controlled
trials. BMJ: British Medical Journal, 327(7429), 1459.
St Pierre, E. A. (2002). Comment: ‘‘Science’’ rejects postmodernism. Educational Re-
searcher, 31(8), 25–27.
Timmermans, S., & Berg, M. (2003). The gold standard: The challenge of evidence-based
medicine and standardization in health care. Philadelphia, PA: Temple University
Press.
Toulmin, S. (2007). Preface: How reason lost its balance. In B. de Sousa Santos (Ed.),
Cognitive justice in a global world: Prudent knowledges for a decent life (pp. ix–xv).
Lanham: Lexington Books.
U.S. Department of Education (2003). Notice of proposed priority: Scientifically based
evaluation methods (RIN 1890-ZA00). Federal Register, 68(213), 62445–62447.
UNAIDS (2011). Terminology guidelines – Revised version, October 2011. Retrieved from
http://www.unaids.org/en/media/unaids/contentassets/documents/unaidspubli-
cation/2011/JC2118_terminology-guidelines_en.pdf
Urban, J. B., Hargraves, M., & Trochim, W. M. (2014). Evolutionary evaluation: Implica-
tions for evaluators, researchers, practitioners, funders and the evidence-based
program mandate. Evaluation and Program Planning. http://dx.doi.org/10.1016/
j.evalprogplan.2014.03.011
Usher, R., Bryant, I., & Johnston, R. (1997). Adult education and the postmodern challenge:
Learning beyond the limits. London: Routledge.
Visvanathan, S. (2005). Knowledge, justice, and democracy. In M. Leach, I. Scoones, & B.
Wynne (Eds.), Science and citizens: Globalization and the challenge of engagement
(pp. 83–94). London: Zed Books.
Walters, P. B., Lareau, A., & Ranis, S. H. (2009). Education research on trial: Policy reform
and the call for scientific rigor. Milton Park, Oxon: Routledge.
Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., et al. (2008).
Bridging the gap between prevention research and practice: The interactive
systems framework for dissemination and implementation. American Journal of
Community Psychology, 41(3–4), 171–181.
Wilson, A. L. (2009). Learning to read: Discourse analysis and the study and practice of
adult education. Studies in Continuing Education, 31(1), 1–12.
Thomas Archibald is an Assistant Professor and Extension Specialist in the Agricul-
tural and Extension Education department at Virginia Polytechnic Institute and State
University (Virginia Tech), where he focuses on program evaluation, evaluation
capacity building, and research-practice integration. He received his doctoral degree
in Education from Cornell in 2013. From 2008 to 2013, he was a Graduate Research
Assistant with the Cornell Office for Research on Evaluation under the direction of
Professor Bill Trochim. Previously, he was a Youth Development Program Manager
with Cornell Cooperative Extension of Tompkins County and an Environmental Edu-
cation Volunteer with the Peace Corps in Gabon.