HANDBOOK OF Complementary Methods in Education Research Judith L. Green Gregory Camilli Patricia B. ElmoreThe American Educational Research Association (AERA) publishes books and journals based on the highest standards of professional review to ensure their quality, accuracy, and objectivity. Findings and conclusions in publications are those of the authors and do not reflect the position or policies of the Association, its Council, or officer Copyright © 2006 by American Educational Research Association All rights reserved. No part of this book my be reproduced in any form, by photstat, microform, retrieval system, or any other means, without prior ‘written permission of the AERA. Lawrence Erlbaum Associates, Inc, Publishers American Educational Research Association 10 Industrial Avenue 1230 Seventeenth Street, NW Mahwah, NJ 07430 Washington, DC 20036-3078 wwierlbaum,com wwieaera.net CIP data can be obtained from the Library of Congress ISBN 0-8058-5932-2 [case] ISBN 0-8058-5933-0 [paper] Books published by Lawrence Erlbaum Associates are printed on acid-free paper, and their bindings are chosen for strength and durability. Printed in the United States of America 098765432Contents INTRODUCTION TO THE VOLUME Preface Lorrie A. Shepard, University of Colorado at Boulder Acknowledgments Introduction to the Handbook: What's Complementary About Complementary Methods? Judith L. Green, University of California, Santa Barbara Gregory Camilli, Rutgers, The Siate University of New Jersey Patricia B. Elmore, Southern Illinois University, Carbondale xi xv xxi Acknowledgment of Advisors and Reviewers 1L_FOUNDATIONS 1 Philosophies of Educational Research Eric Bredo University of Virginia 2 Epistemology and Educational Research Gregory J. Kelly The Pennsylvania State University 3. The Ethics of Educational Research Kenneth A. Strike Syracuse University Il INTRODUCTION TO DESIGN AND ANALYSIS 4__ Analyzing the Multidimensionality of Texts in Education xxv 33 57 75 n Charles Bazenman University of California, Santa Barbaraaa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.CONTENTS: 16 17 18 19 20 21 22 23 24 25 27 Ethnography ‘Kathryn M. Anderson-Levitt University of Michigan—Dearborn Finding Patterns with Field Notes Jan Nespor Virginia Polytechnic Institute and State University Generalizability Theory Richard J. Shavelson Stanford University Noreen M.Webb University of California, Los Angeles Historical Research in Education John L. Rury University of Kansas Historical Studies: Groups/Institutions ‘Annette Henry University of Washington, Tacoma Interviewing in Educational Research Mary E. Brenner University of California, Santa Barbara Introduction to Measurement Theory Linda Crocker University of Florida Item Response Theory ‘Susan Embretson and Xiangdong Yang University of Kansas Linear Regression and Hierarchical Linear Models Delena M. Harrison and Stephen W. Raudenbush University of Michigan Meta-Analysi Gene V Glass Arizona State University The Quantitative Synthesis of Research Findings The Microgenetic Method: Current Work and Extensions to Classroom Research Clark A. Chinn Rutgers, The State University of New Jersey Multiple Methodology in Education Research Mary Lee Smith Arizona State University Narrative Inquiry F. Michael Connelly University of Toronto D. Jean Clandinin University of Alberta vii 279 297 309 323 333 357 371 385 4il 427 439 457 477aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.xi PREFACE stand the kinds of questions that can be addressed by each method and to help seasoned researchers learn new perspectives and skills. The publication of the Handbook of Complementary Methods in Education Research, we hope, puts researchers’ attention on substantive problems with the under- standing that research methods are an important means tothat end. AERA is indebted tothe editors and authors of the Handbook of Complementary Methods in Education Research for their contri- butions to the preparation of the next generation of researchers in education, —Lortie A. Shepard University of Colorado at Boulderaa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.INTRODUCTION TO THE HANDBOOK xvii The principles guiding the process of bringing together different theoretical perspectives and research approaches, however, are not well developed. This argument is represented in a recent Na- tional Forum on Applying Multiple Social Science Research Methods to Educational Problems, co-sponsored by the Center for Education of the National Academies, the American Educational Research Association, the American Psychological Association, and the National Science Foun- dation, In the announcement of this forum, it was argued that “The application, fit, and articulation of different scientific research methods to tackle major issues relevant to educational policy and practice is a largely undeveloped area that is ripe for sustained inquiry and knowledge accumula- tion” (AERA, November, 2004). In framing this forum around these issues, leaders of these major research organizations, each with interests and areas of research in education, suggested that the issue of bringing different tra- ditions together is not solely the task of the individual researcher or research team. It is a problem for the fields that engage in research in and on education, Today, there are no commonly agreed on guidelines for the use of multiple methodologies in particular projects, although some disciplines (eg., health science) have begun to publish handbooks and other guidelines for mixed methods. Whatis evident, however, is that the challenge facing those secking to use different theoretical, not merely alternative methods, is to identify which can be productively brought together—for what purpose(s), in what ways, and on what scale—to explore which phenomena. In this volume of Complementary Methods, these issues are raised in a number of chapters, in- cluding one specifically addressing the design of multiple methods by M. L. Smith, Additionally, when the chapters focusing on programs of research are examined, the impact of different ap- proaches becomes more visible, when the phenomenon under study becomes the focus, not the ‘method itself. In these chapters, the methods are most often used to address different questions and are less frequently used in complementary ways. This observation suggests that the answer to what constitutes complementarity among methods or what relationships need to exist for methods to be defined as complementary, rather than merely mixed, is an area that needs careful discussion and debate. This issue is not a problem for education research alone. Across disciplines, issues of complementarity have been explored for varying lengths of time. A search for definitions of the terms complementary, complementary methods, and complementarity across disciplines shows that these issues have been, and are currently being, debated in a broad range of disciplines in the natural and social sciences, including physics, linguistics, allied medicine, law, environmental studies, psychology, neuroscience, sociology, genetics, organizational studies in business, eco- nomics, mathematics, history, and education, among others. What became evident in our explora tion of these concepts across disciplines is that central to most definitions is the concept of relationships between two or more phenomena, These debates focus on both the complementarity of phenomena under study and methods used to study them, supporting the argument that the- ory-method relationships are critical to understanding and defining whar is complementary about complementary methods. Inthe earlier volumes, as suggested previously, complementary methods referred primarily to alternative methods. As education researchers, we need to explore which of the complex phenom- ena that we are examining or assessing are complementary, and if they are related, what the nature of that relationship is, From this perspective, we need to consider how the phenomena under study are conceptualized, if we are to select appropriate sets of methods to mix and/or use incomplemen- tary ways, To support this direction, the chapters in this volume provide historical and conceptual information about what each method is designed toexplore, the nature of the phenomena involved, and the questions each can address. As such, they lay the foundation for exploring which ap- proaches might be productively brought together to study a common phenomenon, and which might be juxtaposed to make visible similarities, differences and complementarities between phe-aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.On Reading and Using the Volume: Notes to Students As editorial assistants, who had the privilege of working with the editors on this volume, we were asked to write a note to our fellow graduate students—a primary audience for this Handbook of Complementary Methods in Education Research, Throughout our teaching and graduate careers, we have come to understand that like teaching, research is a situated and contextualized learning process, requiring careful consideration of the traditions underlying research methodologies. This volume was designed to provide an introduction to the history, theory, and practice guiding each of the traditions represented in the current Handbook of Complementary Methods in Education Re- search. We see this volumeas both supporting and extending our work as graduate students, and as opening possibilities for new research directions throughout our careers. This new Handbook is a resource that can help us conceptualize, design and expand our research programs, and can help us create and foster dialogues within and across education research traditions. The need for dialogue was captured by Lorrie Sheppard, in her introduction to the Presidential Invited Session for Complementary Methods (AERA 2004). The vision of the editors, AERA pres- idents and various commitiees involved in the publication of this volume was to provide access 10 issues of epistemology associated with the broad range of research traditions currently used by members of AERA. The richness of traditions represented brings unique challenges. since, as the editors state, no one person can know and do research from all traditions. We found that the dia- Jogues among the editorial team from different disciplinary backgrounds were instrumental in bridging the challenge and making visible to us potential links between and among different re- search methodologies and their underlying theories. The need forsuch dialogues in the fieldis cap- tured by Kelly (Chapter 2), who proposes guidelines for respectful and productive dialogue and debate within and across research communities in education, As you read, you will find that like the first two editions, this volume is by no means a “cook- book,” containing easy-to-follow recipes of how to do research using a particular method. Though initially designed for beginning graduate students, the volume in fact presents a broad range of dif- ficulty levels. While all of the chapters are a form of introduction to the field, some will require more background knowledge than others. The volume is resource and areference that can help us explore what is available and whatis possible in education research. It will require us to go to addi- tional resources and to engage in conversations with our professors, mentors, and colleagues within and across institutions. In the remaining section, we share some of the approaches we found helpful in working with this volume as well as the questions we asked in trying to understand the arguments and method- ological frameworks presented by the authors. We hope that these will also be useful to you. ‘* We found it useful to first browse through the whole volume to see what is available, ie., to identify the landscape of research represented. The preface and the introduction were partic- ularly helpful in orienting us to the scope of research traditions in the volume. ‘© The first three foundational chapters provided a lens through which to read any particular chapter and across chapters. These chapters guided us in exploring the questions authors xxiaa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.REVIEWERS 10 uw 12 it: REVIEWERS Philosophies of Educational Research Jim Giarelli, Ruigers, The State University of New Jersey Kenneth Howe, University of Colorado at Boulder jemology and Educational Research Rick Duschl, Rutgers, The State University of New Jersey Linda Tillman, University of North Carolina, Chapel Hill Ey ‘The Ethics of Educational Research Valerie Janesick, Roosevelt University Ivor Anton Pritchard, Institute of Education Sciences, U.S. Department of Education. Analyzing the Multidimensionality of Texts in Education Cynthia Lewis, University of Minnesota Peter Smagorinsky, The University of Georgia Arts-Based Educational Research Susan Finley, Washington State University, Vancouver Donald Blumenfeld-Jones, Arizona State University Case Study Methods Robert Stake, University of Ilinois, Urbana~Champaign Christopher M. Clark, University of Delaware Cross-Case Analysis Vivian Gadsden, University of Pennsylvania Joseph Tobin, Arizona State Univers Curriculum Assessment Ginger Weade, Ohio University Gerunda B. Hughes, Howard University Data Modeling: Structural Equation Modeling Ralph O. Mueller, The George Washington University Janet K, Holt, Northern Illinois University Definition and Analysis of Data from Videotape: Some Research Procedures and Their Rationales Carol Dixon, University of California, Santa Barbara Tim Koschmann, Southern Illinois University. Springfield Design Experiments Lauren Resnick, University of Pittsburgh Richard Lehrer, Vanderbilt University Developmental Research: Theory, Method, Design and Statistical Analysis Alison Louise Bailey, University of California, Los Angeles Kathryn V. Drummond, American Institutes for Research Susan Goldman, University of llinois, Chicago Discourse-in-Use James P. Gee, University of Wisconsin, Madison Jerri Willett, University of Massachusetts, Amherstaa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.FOUNDATIONS The three chapters in this section lay a foundation for readers seeking to understand the underlying philosophical bases of different theories guiding research in education and their consequences for knowledge construction within the research community. The first chapter, Philosophies of Educational Research (Eric Bredo), provides an historical overview of the range of traditions that have influenced educational research in the past and how these traditions con- tinue to shape developments within the field of education. The second chapter, Epistemology and Educational Research (Gregory Kelly) provides a framework for dialogue and debates among the different traditions, and how these dialogues can lead to the construction of legitimate knowl- edge in education, The third chapter, The Ethics of Educational Research (Kenneth Strike), raises key questions about the ways in which the educational research community can enhance individual and social welfare with quality research. Together, these chapters lay a foundation for future debates and deliberations about standards for research, contributions of different research traditions, questions that can be addressed by each, and approaches to the generation of relevant knowledge. The chapters raise ontological, epistemological and ethical issues necessary for the con- structive exchange of ideas about the nature, purposes and approaches to research in education. For this reason, readers unfamiliar with the ongoing developments in the fields of philosophy in education, philosophy of science, sociology of knowledge and ethics will find these chapters informative, yet, at times, challenging. Together, the chapters provide a foundation for under- standing how education as a discipline informs educational progress, processes, practices, and policymaking. ‘The authors provide a rich explication of theory-method issues involved in under- standing and conducting research in education. These chapters invite multiple readings to explore the new terrains and directions raised by the authors,aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.6 BREDO 1 have tried to handle this by considering rough historical phases within each family of thought, acknowledging that discussion within a family has frequently ended up challenging assumptions with which it began. All of this oversimplifies things greatly because there has been much dis- cussion and borrowing across traditions. It also oversimplifies in tracing an approach to a par- ticular individual who may more appropriately be viewed as stating a position felt by many at the time. EXTERNAL RELATIONS The dominant tradition of educational and social science research has been an empiricist philos- ophy based on extemalist assumptions. In the late 1950s, for example, the psychologist Lee J. Cronbach argued that there were two disciplines of scientific psychology. Experimental psy- chologists (e.g., behaviorists) studied the effects of environmental contingencies on behavior regardless of individual differences. On the other hand, personality psychologists (e.g., 1Q testers) studied effects of individual differences regardless of environmental contingencies (Cronbach, 1957). Cronbach argued that these two rather separate “disciplines” of psychology could be integrated by adopting a more general approach that focused on the interactive effects of personality and environment on behavior. Each of the one-sided approaches could be viewed as a special case of 4 more general interactionist model in which the interactive effects of one or another factor might be strong or weak as the case might be. Ina way, Cronbach’s analysis incorporated all three orientations considered here. Behaviorists were “externalists,” personality psychologists “intemnalists,” and Cronbach’s position represented a third “interactionist” alternative, What was less evident at the time was that all three approaches agreed on what one might call methodological extemnalism. All adopted an externally oriented empiricist or positivistic approach to research, seeking to induce general laws from observations or data viewed as given independently of the laws or generalizations being derived. This extemalist consensus fell apart in the decades following Cronbach’s article in which “positivism” became a “term of abuse.” as Denis Phillips (1983) noted. Suffering criticism within its own ranks as well as from outsiders, positivism died as a philosophy of science, to be replaced in recent years by an emerging “post-positivistic” philosophy (Phillips, 1983). It may help to consider a brief history of this line of thought to understand better the relations between these various “isms” (empiricism, positivism, post positivism) and to see what has died and what has replaced it. Empiricism Empiricist philosophies of knowledge generally argue that knowledge is based in direct experi- ence of concrete objects or events. The English philosopher, John Locke (1632-1704), one of the principal founders of empiricism, argued that all knowledge comes from “experience” (Locke, 1974), Sensory experience of extemal things occurs when “particular sensible objects, do con- vey into the mind several distinct perceptions of things, according to those various ways wherein those objects do affect them” (Locke, 1974). In other words, external objects stimulate the sen- sory organs conveying basic information to the mind, which then combines, generalizes, and abstracts these “elementary ideas” to form more complex and general knowledge of objects and their relationships. But if all ideas come from experience, and people have different experiences, how do they come to agree on things? Locke's answer was that the “primary” qualities of things, such as their physical solidity, shape, size, or movement, are directly and accurately perceived “as they are in a mirror.” while “secondary” qualities, like their taste or color, are more variable and subjective (Ibid., 1974,aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.aa You have either reached a page that is unavailable for viewing or reached your viewing limit for this book.10 BREDO it emerged (Goodman, 1972; Hanson, 1958; Kuhn, 1962; Lakatos & Musgrave, 1970; Popper, 1959; Quine, 1953). The view of science coming from this criticism is sometimes referred to as, “ ivism.” While this is often taken as a new and distinctive philosophy of science it might be better regarded as a diverse set of arguments against positivism that are still being consolidated. ‘The arguments against logical positivism cannot be considered in detail here, but four gen- eral points summarize their gist. First, the logical positivists' strong distinction between fact and theory, implicit in the notion that observational facts are logically independent of the theories they test, came in for question. As the philosopher Willard van Orman Quine (1953) pointed out, the best of modem science sometimes bends theoretical rules, even the rules of formal logic, to suit its practical purposes. Quantum mechanics rejected the “law of the excluded middle,” for example, the notion that a statement must be either true or false, and not both (Quine 1953, p. 43). Even the most universal and abstract theoretical considerations are subject to “empirical” influence in the best of modern science. Quine also questioned the notion that meaningful asser- tions can be reduced to the elementary perceptual experiences verifying them. He suggested that beliefs function in networks rather than individually, raising questions about which belief a given factual observation verifies or falsifies. Because a whole set of beliefs function together as a “web” that may be adjusted to accommodate a new fact in various ways, pragmatic considera tions, such as the usefulness of a given set of beliefs for the purposes at hand, frequently dictate how one resolves an issue, rather than formal logic or observational fact per se. This suggests that the positivists were too dogmatic about the absoluteness of both formal logic and observa tional fact, each of which is often interpreted in different ways in practice to suit the purposes at hand. As a result we might say, today, that facts are “theory laden” (or theories “fact laden”), We use particular and general statements to help interpret one another. ‘A second line of criticism questioned the notion that scientific laws can be “verified.” The problem with induction, as Karl Popper and others argued, is that just because a pattern of events has been observed in the past gives no logical reason to suppose that things will happen simi- larly in the future. The sun has appeared regularly every morning but some moming it may not. ‘This led to the suggestion that scientific laws can never be verified, only falsified (Popper, 1959). Approached in this way, what makes a claim “scientific” is not the fact that it has been “veri fied,” but rather the fact that it is at least potentially falsifiable by empirical observations and has been subject to stringent attempts at such disproof. This conception of science is not so dogmatic as the positivistic account, because in it all scientific claims remain falsifiable by future events rather than asserted as definitely “verified.” (Popper's approach is sometimes termed “critical rationalism” because it focuses critical attention on the beliefs from which claims are deduced.) While one may not accept Popper's assertion that there is no logic of induction, only one of deduction (Langley, Simon, Bradshaw, & Zytkow, 1987; Peirce, 1923b), his emphasis on criti ally testing hypotheses hy looking for disconfirming evidence, and not merely confirming evi dence, is important for educational researchers who sometimes look only for data confirming their views. A third line of criticism objects to the overly individualistic conception of science implicit in the empiricist and positivistic tradition. Thomas Kuhn’s work, in particular, gave new empha- sis to the notion that science is a social enterprise conducted by a community of inquirers oper- ating with their own conventions (Kuhn, 1962). Viewing science as a collective enterprise not only highlights the way scientists seek to replicate, test, and challenge each other's results, but also suggests that the norms and background assumptions adopted by those in the scientific com- munity affect the course of science. Since any set of data may, in general, be explained in many different ways, then the taken-for-granted assumptions of the scientific community may affect the way the results are interpreted (e.g., the assumption that nature is uniform). What is taken as