You are on page 1of 4

EDUCATIONAL PSYCHOLOGIST, 39(4), 199–201

Copyright © 2004, Lawrence Erlbaum Associates, Inc.


SANDOVAL AND BELL
INTRODUCTION

Design-Based Research Methods for


Studying Learning in Context: Introduction
William A. Sandoval
Graduate School of Education and Information Studies
University of California, Los Angeles

Philip Bell
Cognitive Studies in Education
University of Washington

The field of psychology has a long history of interaction An educational psychology that is both usable in a practi-
with education, and educational psychology has had a pro- cal sense and scientifically trustworthy cannot proceed with-
found impact on how issues of learning have been framed out directly studying the phenomena it hopes to explain in its
and studied in educational contexts. Still, it has never been inherent messiness. A little over a decade ago, Brown (1992)
simple to translate theoretical insights into educational described her evolving approach to “design experimenta-
practice. Educational psychology has been criticized for not tion” as an effort to bridge laboratory studies of learning with
creating “usable knowledge” (Lagemann, 2002). Currently, studies of complex instructional interventions based on such
educational researchers generally have been pushed to jus- insights. She showed how insights from the laboratory were
tify how their claims are “scientific” and “evidence-based” inherently limited in their ability to explain or predict learn-
(National Research Council, 2002). There is a tension be- ing in the classroom. The challenge, as she saw it, was to de-
tween the desire for locally usable knowledge on the one velop a methodology of experimenting with intervention de-
hand and scientifically sound, generalizable knowledge on signs in situ to develop theories of learning (and teaching)
the other. Lagemann, for example, argued that the tradi- that accounted for the multiple interactions of people acting
tional paradigm of psychology has striven for experimental in a complex social setting. At the same time, Collins (1992)
control at the expense of fidelity to learning as it actually was putting forth a notion of educational research as a “de-
occurs. Thus, although such claims might be scientific in sign science,” like aerospace engineering, that required a
one sense, they do not adequately explain or predict the methodology to systematically test design variants for effec-
phenomena they purport to address. This critique extends tiveness. Achieving such a design science, however, requires
the long-standing debate surrounding the ecological valid- a sufficient understanding of the underlying variables at all
ity of well-defined psychological tasks and their relation to relevant layers of a complex social system (schooling)—an
psychological phenomena as they come to occur in every- understanding that we do not yet have (Collins, Joseph, &
day settings (Brunswik, 1943; Lewin, 1943). As a field, we Bielaczyc, 2004).
still lack an adequate methodological reconciliation that at- The last 12 years have seen an increasing uptake of the de-
tends to issues of both experimental control and ecological sign experimentation methodology, so much so that a recent
validity. At the same time, there is considerable unease with handbook on research in math and science education is re-
the perceived “credibility gap” (Levin & O’Donnell, 1999) plete with examples and formulations of the approach (Kelly
of much of educational research because it is not produced & Lesh, 2000). The general approach has been called by
with what are considered to be scientific methods. From many names. We have settled on the term design-based re-
this perspective, the knowledge from educational research search over the other commonly used phrases “design exper-
has limited usability because it is not trustworthy. imentation,” which connotes a specific form of controlled ex-
perimentation that does not capture the breadth of the
approach, or “design research,” which is too easily confused
Requests for reprints should be sent to William A. Sandoval, Graduate
School of Education and Information Studies, University of California, Box
with research design and other efforts in design fields that
951521, 2339 Moore Hall, Los Angeles, CA 90095–1521. E-mail: lack in situ research components. The approach to research
sandoval@gseis.ucla.edu described in this issue is design based in that it is theoreti-
200 SANDOVAL AND BELL

cally framed, empirical research of learning and teaching Hoadley begins by framing perhaps the central issue of
based on particular designs for instruction. Design-based re- any research—the basis on which claims can be warranted.
search simultaneously pursues the goals of developing effec- He discusses how the interplay between designing and then
tive learning environments and using such environments as studying interventions in naturalistic settings can lead to
natural laboratories to study learning and teaching. On the re- “methodological alignment.” As initially unpredicted obser-
search side of the endeavor, design-based researchers draw vations arise among predicted ones, a design-based research
from multiple disciplines, including developmental psychol- team’s methodological approach changes with developing
ogy, cognitive science, learning sciences, anthropology, and theoretical knowledge, leading to intervention designs that
sociology. On the design side of the work, researchers draw are better fit to their intended setting and to better explana-
from the fields of computer science, curriculum theory, in- tions of how they work. Hoadley frames this notion of meth-
structional design, and teacher education. odological alignment in comparison to typical experimental
Since Brown’s and Collins’influential works, the paradigm and quasi-experimental designs of educational psychology to
has evolved primarily as a means for studying innovative separate the methodological threats to validity and reliability
learning environments, often including new educational tech- faced by any research approach from the issues that specifi-
nologies or other complex approaches, in classroom settings. cally face design-based researchers.
As this form of interventionist research has spread, questions Following this are three articles that each take a different
have emerged for which there are not yet clear answers. What slice on how design-based research can contribute to theoret-
exactly counts as design-based research? What kinds of ical understanding of learning in complex settings. Each of
knowledge can design-based research produce? What stan- the articles by Sandoval, Tabak, and Joseph reveal how the
dards do, or should, exist to judge the quality of design-based design of complex interventions is an explicitly the-
research? Such questions are being asked by both practitioners ory-driven activity. Sandoval introduces the notion of “em-
of the approach and observers of educational research (see bodied conjectures” to characterize how instructional de-
Kelly, 2003). The articles in this special issue of Educational signs materially embody theoretical conjectures about how
Psychologist extend recent discussions of these questions, people learn. They therefore carry expectations about how
both to situate design-based research within a broader context designs should function in a setting, and tracing how such ex-
of research on learning and to continue a needed conversation pectations are met or unmet can refine the underlying theo-
on the nature of design-based research specifically and useful retical conjecture. Through a retrospective analysis, he ar-
forms of educational research generally. gues that an important way of increasing the rigor of
The articles in this set arise from work supported by an Ad- design-based research is for researchers to explicitly map the
vanced Studies Institute grant from the Spencer Foundation to embodiment of particular conjectures through their design
a group of early career researchers trained in design-based re- reification and to then design research studies to specifically
search methods. Each article in the set addresses various as- tests the predictions that result. Such predictions pertain to
pects of one central question: How does the effort to design both outcomes expected from the intervention and ways in
complex interventions influence research? The effort to de- which designed scaffolds are expected to function. The need
sign complex interventions raises a set of methodological and to link outcomes to these expected functions across research
theoretical issues. One of the most commonly faced method- iterations is the source of power from this analytic approach.
ological issues in design-based research is the tension between Tabak considers the theoretical and methodological ten-
making an intervention “work” in a complex setting, which of- sions that arise when complex interventions are introduced
ten necessitates changing the intervention as it unfolds (in a into classroom settings. She describes how intangible aspects
way that directly mirrors the dynamic, contingent nature of de- of interventions, such as designs for particular forms of class-
cision making during teaching), with the researchers’need for room discussions, blur the boundaries between the interven-
empirical control, which argues against changing the planned tion and what is typically thought of as “the context.” Rather
“treatment.” The general issue this raises is one of causal attri- than consider such a blurring as only a disadvantage, Tabak ar-
bution: What makes a particular intervention successful in a gues that attention to such emergent activity structures is a key
particular place? How can what is learned from a particular element of design-based research and can contribute to the de-
success be generalized? It has been argued that design-based velopment of theories of contextualization. Her argument is an
research can develop different kinds of knowledge, including explicit bridge between psychological perspectives that
better theoretical understanding of the learning phenomena would, under the guise of experimental control, generally ig-
addressed by an intervention and knowledge of useful and nore aspects of context outside of the perceived intervention
generalizable design practices (Design-Based Research Col- treatment and anthropological views of context that do not at-
lective, 2003; Edelson, 2002). The set of articles in this issue tempt to distinguish between “native” and designed features.
describe how the design-based research methodology can pro- Joseph describes how the complexity involved in devel-
duce such knowledge by raising issues that arise from the in- oping a novel instructional approach and trying to under-
fluence of design on research and how design-based research- stand its enactment and potential benefits forced choices
ers try to address them. about which aspects of the intervention became the focus of
INTRODUCTION 201

research and which aspects were just “engineered” to work As a collection, these articles depict the unique, evolving,
in the immediate setting. This is a common occurrence dur- and expanding role of design-based approaches to educa-
ing design-based research efforts, and such choices arise tional inquiry. They also highlight the pressing issues and fu-
somewhat unpredictably from the setting of an intervention. ture directions currently facing researchers making use of
Joseph describes how such choices can be theoretically this methodological orientation. We believe that by bringing
guided and thus contribute to theoretical refinement. In her some much-needed clarity around the nature of design-based
case, the effort to design an interest-based curriculum for research to the broader readership of Educational Psycholo-
elementary students exposed the limitations to current theo- gist, it will fuel productive discussions of research methodol-
retical ideas of motivation and interest and led to the devel- ogy and modes of inquiry currently appropriate for the study
opment of a framework for conceptualizing interest in a of learning.
“usable” way.
Bell ends the article set by laying out what he refers to
REFERENCES
as the “grammar and epistemology” of design-based re-
search. In surveying design-based research efforts from the Brown, A. L. (1992). Design experiments: Theoretical and methodological
last decade, Bell identifies several variations of the method- challenges in creating complex interventions in classroom settings. Jour-
ology that differ in their theoretical perspective and argues nal of the Learning Sciences, 2(2), 141–178.
that these differences in underlying theoretical position lead Brunswik, E. (1943). Organismic achievement and environmental probabil-
to epistemological differences in the kinds of claims that ity. Psychological Review, 50, 255–272.
Collins, A. (1992). Toward a design science of education. In E. Scanlon & T.
design-based researchers might try to make and the bases O’Shea (Eds.), New directions in educational technology (pp. 15–22).
on which they warrant such claims. He argues for the im- New York: Springer-Verlag.
portance of recognizing design-based research as encom- Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoreti-
passing this variety of theoretical and epistemological per- cal and methodological issues. Journal of the Learning Sciences, 13(1),
spectives, to both better understand just what design-based 15–42.
Design-Based Research Collective. (2003). Design-based research: An emerg-
research is (or might be) and clarify how design efforts can ing paradigm for educational inquiry. Educational Researcher, 32(1), 5–8.
be better tied to theoretical and methodological consider- Edelson, D. C. (2002). Design research: What we learn when we engage in
ations. In the end, he argues against a singular definition of design. Journal of the Learning Sciences, 11(1), 105–121.
what might constitute “scientific” design-based research Kelly, A. E. (Ed.). (2003). Special issue on the role of design in educational
and for a theoretical, and hence methodological, pluralism research [Special issue]. Educational Researcher, 32(1).
Kelly, A. E., & Lesh, R. A. (Eds.). (2000). Handbook of research design in
to efforts that seek to understand learning and influence ed- mathematics and science education. Mahwah, NJ: Lawrence Erlbaum
ucational practice. Associates, Inc.
In her commentary, O’Donnell considers the methodolog- Lagemann, E. C. (2002). An elusive science: The troubling history of educa-
ical issues and approaches sketched by the article set in terms tion research. Chicago: University of Chicago Press.
Levin, J. R., & O’Donnell, A. M. (1999). What to do about educational re-
of ideas of quasi-experimental and experimental research de-
search’s credibility gaps? Issues in Education, 5, 177–229.
sign. She critically examines the assumptions about de- Lewin, K. (1943). Defining the “field at a given time.” Psychological Re-
sign-based research behind each article and pushes on meth- view, 50, 292–310.
odological issues that remain outstanding and hence a threat National Research Council. (2002). Scientific research in education. Wash-
to the promise of design-based research. ington, DC: National Academy Press.