Understanding and Interpreting Educational Research
Visit the link below to download the full version of this book:
[Link]
arch/
Click Download Now
© 2013 The Guilford Press
A Division of Guilford Publications, Inc.
72 Spring Street, New York, NY 10012
[Link]
All rights reserved
No part of this book may be reproduced, translated, stored in a retrieval
system, or transmitted, in any form or by any means, electronic, mechanical,
photocopying, microfilming, recording, or otherwise, without written permission
from the publisher.
Printed in the United States of America
This book is printed on acid-free paper.
Last digit is print number: 9 8 7 6 5 4 3 2 1
Library of Congress Cataloging-in-Publication Data is available
from the publisher.
ISBN: 978-1-4625-0962-1 (paperback)
ISBN: 978-1-4625-0974-4 (hardcover)
Portions of this book are drawn from Research Methods: Learning to Become a
Critical Research Consumer, published by Pearson Education, Inc.
Preface
Objectives of This Book
Our primary purpose in writing this book is to provide a comprehensive coverage
of research design issues that students find throughout the educational and psycho-
logical research literature. For example, group designs are found in psychological and
educational research, but other designs, such as qualitative methods and single-case
research designs, are also common. It is critical for readers to understand our approach
to these different methods. We do not discount the importance of one or more meth-
ods, or claim that one type of methodology is superior to another. Our approach is
that there are several different types of research methodologies. The choice of meth-
odology should be based not on some preconceived notion of the superiority of one
over another but on an assessment of which methodology is best suited to answer the
research question or to test the hypothesis at hand. Therefore, the selection of a partic-
ular methodology should be based not on personal bias but on a thorough knowledge
of the different methodologies.
Few other research methods books provide the depth of coverage of the wide
array of designs covered here. For instance, whereas most research methods books
provide superficial coverage of single-case designs as part of a chapter, we provide
three chapters on these important designs, which are prevalent in behavioral and spe-
cial education research. Researchers may find the coverage of these designs particu-
larly useful when conducting field studies in which experimental control is required.
However, we do not provide expanded coverage of one methodology at the expense
of other approaches. We attempt to provide equal depth of coverage of the common
methods found in the research literature, including the latest research techniques on
mixed-methods research, grounded theory development, phenomenology, and con-
ducting literature reviews including online searches. We have also fully covered sur-
vey research, as a stand-alone chapter in the quantitative research methods section of
the text.
v
vi Preface
Organization of the Book
The organization of this book reflects our primary goal in writing it: to aid students in
becoming consumers of research. In our opinion, before a student is ready to conduct
independent research, he/she needs an understanding of the research process, includ-
ing the decisions researchers make and why they make them. Therefore, we placed
material on variability and threats to internal and external validity near the begin-
ning of the book (Chapter 2). This material is typically located much later in other
textbooks. However, given that variability, and the control of variables that contribute
to variability, is so critical in the research endeavor, placing the material early on is
more suitable.
We have also attempted to reduce the redundancy found in many books by pre-
senting information in a sequenced format, the material in each chapter building on
material covered in previous chapters. Therefore, it is best to present the material in
the book in the sequence provided. Students will find that the sequence is logical and
will aid in their mastery of the concepts.
Among the more important aspects of the book are the sample articles at the end
of each of the research design chapters. Since the main purpose of our book is to teach
students how to consume and understand research, these are provided to give stu-
dents opportunities to practice interpreting an article that covers techniques discussed
in the chapter. Each article is followed by questions to prompt class discussion of the
article and guide students through a critique of how the research was conducted, why
it was conducted that way, and whether the researchers’ conclusions were warranted.
It is important for students to read these articles and to complete the activities associ-
ated with them to enhance their learning.
Topic Coverage
The book includes seven major parts comprising 16 chapters, reflecting the fact that
many universities cover research methods as a 16-week course. Following is a descrip-
tion of the contents associated with each of the parts.
Part I discusses skills in learning how to become a critical research consumer
(Chapter 1). Information needed to think critically about research includes descrip-
tions of science, what science is, the process of science, and the purpose of science.
Part II covers critical issues in research, including the concept and sources of
variability. Internal and external validity are keys to understanding much of the edu-
cational and psychological research produced. Therefore, a great deal of time is spent
on these concepts. Statistical and social validity are also described. A discussion of
the differences in these important forms of validity is presented (Chapter 2). The con-
cepts of reliability and validity from quantitative and qualitative perspectives are also
discussed in Part II. Interobserver agreement, a special form of reliability and valid-
ity that is used in single-case research, is presented. Finally, factors to consider when
assessing measurement devices in the context of an investigation are discussed (Chap-
ter 3). Several examples and nonexamples are presented throughout the chapter to
help students learn key concepts.
Part III concerns quantitative research methods. Part III begins with a presenta-
tion of some basic statistical concepts. Knowledge of these concepts is critical for an
understanding of many quantitative designs. Sampling methods are also discussed at
Preface vii
length. The method of sampling is important in determining the type of design used,
as well as claims that can be made about the generalizability of the results (Chapter
4). The different methodologies are covered, including true experimental, factorial,
quasi-experimental, and preexperimental designs (Chapter 5). Additionally, causal–
comparative research methods (Chapter 6), correlational research methods (Chap-
ter 7), and survey research (Chapter 8) are presented. At the end of each chapter, we
describe when to use each form of research method. Finally, Chapters 5–8 present
sample articles for students to critique, including rating forms to analyze various
aspects of the studies.
Part IV focuses on qualitative research methods (Chapter 9) and data collection
and designs (Chapter 10). Methods of sampling are discussed. Chapter 10 closes with
a discussion about when different qualitative research methods should be used. A
sample article for students to analyze and an analysis form are included in the chap-
ter.
Part V presents a detailed discussion of single-case methods. We show how
single-case methods may be thought of as a combination of quantitative and qualita-
tive designs. Methods of graphing and withdrawal and associated designs are dis-
cussed in Chapter 11. Multiple-baseline designs (Chapter 12) and additional single-
case designs (Chapter 13) are presented. Each chapter discusses when to use the
various types of single-case designs. A sample article for critique, along with a rating
form, is presented at the end of each of the chapters.
Part VI focuses on evaluation research, including program evaluations (Chapter
14) and a detailed discussion of research syntheses (Chapter 15). We discuss when
each form of research is conducted. We also present a sample article for critique at the
end of each chapter.
Part VII covers issues to consider when planning and conducting a research proj-
ect (Chapter 16). Recall that the book is designed to teach students to think critically
about research; however, we also want to produce scientist-practitioners. Practitioners
can, and in some instances should, produce data to make informed decisions. Practi-
tioners who produce such data are engaging in action research. A discussion of how
to conduct action research is presented. When a decision is made to conduct an inves-
tigation, ethical issues must then be considered. These ethical issues are described.
“Action researchers” should also know the process one goes through to write and
submit an article for publication. We describe that process. Finally, a sample article is
presented at the end of the chapter to critique.
Pedagogical Features
This book has several pedagogical features to aid instructors with material presenta-
tion and to help students understand the multitude of concepts they encounter. Fol-
lowing is a brief description of the major features of our book:
• Objectives. Each chapter begins with a numbered list of objectives. These serve
as an anticipatory set allowing students to obtain a quick snapshot of the con-
tents of the chapter.
• Chapter-opening graphic organizers. Each chapter contains a graphic orga-
nizer. These enable students to see the contents of each chapter in an organiza-
tional framework.
viii Preface
• Overviews. An overview provided at the beginning of each chapter allows stu-
dents to understand the importance of the material that follows.
• Main headings in the form of questions. All main headings are framed as
questions in order to facilitate student note taking. Students should be taught
how to list each question heading, then to read the materials under each head-
ing to answer the question. This approach is important in helping students bet-
ter understand chapter content.
• Figures and tables. Several illustrations and tables in each chapter present a
visual representation of concepts or provide a summary of critical material.
• Running glossary. In each chapter, a running glossary box defines key terms
introduced on a two-page spread.
• Research examples. Throughout each chapter, research examples highlight
key concepts. Additionally, each research methods chapter contains boxes with
research questions. Descriptions of investigations that answer these questions
are provided throughout the chapters. These boxes are designed to prompt stu-
dents to think about how they would answer the questions and then provide a
description of how researchers would answer them. These activities give stu-
dents a way to evaluate their understanding of the material.
• Bulleted summary. Each chapter ends with a bulleted summary. These sum-
maries serve as a review of material and a study aid for tests and quizzes.
• Discussion questions. Questions provided at the end of each chapter allow for
rich discussion of chapter material. These questions also help students evaluate
their understanding of concepts presented throughout the chapter.
• Additional exercises. Additional (i.e., interpretation, practice) exercises are
provide in Chapters 2 and 3 to highlight important concepts needed as a foun-
dation for later material.
• Journal articles as illustrative examples. Each research methods chapter
(Chapters 5–8 and 10–16) provides an illustrative example from published peer-
reviewed articles. These examples give students opportunities to read authen-
tic research articles and to critique them with guidance from the instructor.
This feature is perhaps the one that best exemplifies the purpose of the book—
to teach students how to read and consume research. Each illustrative exam-
ple has questions to prompt discussion of the article and activities to guide
students through the critiquing process. For example, several chapters include
threats to internal validity and external validity forms.
• Additional research examples. Lists of additional research examples are pro-
vided in Chapters 5–8 and 10–16 for instructors or students who want to access
further readings on the concepts within each chapter.
• End-of-book glossary. The book includes a detailed glossary of key terms. This
glossary provides an additional resource for students to access when defining
terms.
• Complete list of references. A complete list of references is found at the end of
the book, as opposed to the end of each chapter. The placement of the references
makes it easier for instructors and students to find cited material.
• Author index. An author index at the end of the book aids in the search of par-
ticular authors of interest to students.
• Subject index. A subject index at the end of the book helps students locate spe-
cific information in the book.
• Supplementary electronic instructor’s resource manual. This comprehensive
Preface ix
aid for course preparation includes a detailed outline of each chapter for lecture
notes (also available in downloadable PowerPoint format); answers to within-
text discussion questions, interpretation/practice exercises, and illustrative
example questions; and an extensive test bank of multiple-choice questions and
short-answer/essay questions readable to most test management software. A
link to this electronic manual is sent automatically via e-mail to any professor
who orders an exam copy directly from The Guilford Press. Please contact cus-
tomer service (info@[Link]) for further information.
Audience for the Book
The book is designed to be used across several different levels of instruction. The pri-
mary audience for the book is upper-level undergraduate or master’s students. How-
ever, the book may also be used (and has been used in the past) at the doctoral level
with supplementary materials. With this in mind, we attempted to use language that
is appropriate not only for advanced undergraduate students but also for students at
more advanced levels of study.
Given the sequencing of the material in the book, it may also be used for self-study.
We have had success in the past with students studying the material in the book on
their own. If the book is used for self-study, we recommend reviewing the Objectives to
each chapter, answering the Discussion Questions at the end of each chapter, and also
completing any additional exercises that are available. It is especially important for the
self-study student to read the sample articles at the end of each research design chapter
carefully and to complete the indicated activities.
Acknowledgments
We would be remiss not to acknowledge the individuals who have affected our edu-
cational careers. First and foremost, we wish to thank our families for their support.
We would especially like to thank our children for allowing us to take the time to
complete the project.
In order to complete this text, several individuals were involved. We wish to
thank everyone at The Guilford Press for their continued support of the entire project,
especially C. Deborah Laughton, without whom this project would not have come to
fruition. We would also like to thank Ryan Sain for taking the lead on developing the
instructor’s manual. Finally, we are grateful to the reviewers whose criticisms were
highly constructive: Melody Whiddon, College of Education, Florida International
University; Anthony Salvatore, Department of Speech–Language Pathology, Univer-
sity of Texas, El Paso; Tracy Walker, School of Liberal Arts and Education, Virginia State
University; Hisako Matsuo, Department of Sociology and Anthropology, Saint Louis
University; and Hansel Burley, College of Education, Texas Tech University.
Brief Contents
Part I Under standing Rese arch
1 Thinking Critically about Research 3
Part II Critic al Issues in Rese arch
2 Fundamental Issues for Interpreting Research 39
3 Reliability, Validity, and Interobserver Agreement 69
Part III Quantitative Rese arch Methods
4 Basic Statistical Concepts and Sampling Procedures 99
5 Experimental Designs 135
6 Causal–Comparative Research 176
7 Correlational Research 207
8 Survey Research Methods 256
Part IV Qualitative Rese arch Methods
9 Basic Understandings in Qualitative Research 293
10 Data Collection and Designs in Qualitative Research 323
xi
xii Brief Contents
Part V Single- C a se Rese arch Methods
11 Withdrawal and Associated Designs 369
12 Multiple‑Baseline Designs 408
13 Additional Single‑Case Designs 450
Part VI Evaluation Rese arch
14 Program Evaluation 501
15 Evaluating the Literature 528
Part VII Action Rese arch
16 Action Research: Moving from Critical Research Consumer 579
to Researcher
Glossary 621
References 633
Author Index 645
Subject Index 653
About the Authors 665
Extended Contents
Part I Under standing Rese arch
1 Thinking Critically about Research 3
Objectives 3
Overview 4
How Important Is Critical Thinking? 4
What Are the Elements to Becoming a Critical Research Consumer? 5
What Are Science and the Scientific Method? 6
What Are the Purposes of Science? 7
Description 7
Box. Predictions Gone Awry 8
Prediction 8
Improvement 9
Explanation 9
Box. What Works Clearinghouse (WWC) 9
Box. Causation or Correlation? 10
What Is a Scientific Theory? 10
Box. Science or Pseudoscience? 12
“Good” versus “Bad” Scientific Theories 12
Testability of a Theory 13
What Are the Types of Scientific Logic? 13
Inductive Logic 13
Box. Examples of Inductive Logic 14
Deductive Logic 15
Box. The Principle of Deductive Logic 16
Combination of Logic Forms 16
Critique of Both Deduction and Induction 17
In What Ways Do We Gain Information? 17
Tenacity 18
Intuition 18
Authority 19
Empiricism 19
Box. Why Question Authority? Why Not? 19
Rationalism 20
What Are Constraint Levels in Educational and Psychological Research? 20
Experimental Research 21
Causal–Comparative Research 23
Box. Examples of Correlation Research Questions in Education 24
Correlational Research 24
Case Study Research 24
Naturalistic or Descriptive Research 25
xiii
xiv Extended Contents
What Are the Differences between Basic and Applied Research? 26
Basic Research 26
Applied Research 26
What Is Replication Research? 27
Reasons for Replication 27
Box. Cold Fusion 28
Types of Replications 31
Replication as the “Big Picture” 33
Summary 33
Discussion Questions 34
Part II Critic al Issues in Rese arch
2 Fundamental Issues for Interpreting Research 39
Objectives 39
Overview 39
What Is Variability? 40
Sources of Variability 40
What Is Internal Validity and Its Threats? 42
Threats That Result in Differences within and between Individuals 43
Threats That Result in Similarities within and between Individuals 50
What Is External Validity and What Are Its Threats? 52
Population Validity 52
Ecological Validity 56
What Are Statistical and Social Validities? 63
Statistical Validity 63
Social Validity 64
Summary 65
Discussion Questions 65
Interpretation Exercises 66
3 Reliability, Validity, and Interobserver Agreement 69
Objectives 69
Overview 70
What Are Reliability and Validity Issues in Quantitative Research? 71
Box. Inappropriate Inferences Made in the Area of Intelligence 71
How Do Researchers Assess the Reliability of Measurement Devices? 72
Types of Reliability Coefficients 72
Standard Error of Measurement 78
How Do Researchers Determine the Validity of Measurement Devices? 79
Types of Validity 80
What Are Reliability and Validity Issues in Qualitative Research? 83
Reliability Issues 84
Validity Issues 84
Illustrative Example of Reliability and Validity Procedures 84
What Is Interobserver Agreement? 86
What Are the Methods of Establishing Interobserver Agreement? 86
Measures of Permanent Products 87
Event Recording 88
Latency and Duration Recording 88
Interval Recording 89
What Are the Factors That Influence Interobserver Agreement? 91
Reactivity 91
Observer Drift 91
Extended Contents xv
Complexity of the Measurement System 92
Observer Expectations 92
What Are the Factors to Consider When Assessing Measurement Devices
in the Context of a Study? 92
Description of the Measurement Device 92
Adjustments to the Measurement Device 92
Appropriateness of the Measurement Device 93
Cooperation of Participants 93
Summary 93
Discussion Questions 95
Practice Exercises 95
Part III Quantitative Rese arch Methods
4 Basic Statistical Concepts and Sampling Procedures 99
Objectives 99
Overview 100
What Are the Scales of Measurement? 100
Nominal Scale 100
Ordinal Scale 100
Interval Scale 101
Ratio Scale 101
What Is a Univariate Frequency Distribution? 101
How Can We Describe Data Sets? 101
Measures of Central Tendency 102
Measures of Variation 103
Shape of the Distribution 104
What Role Do Hypotheses Play in Research? 105
Inferential Statistics 105
The Role of Hypotheses 105
Null and Alternative Hypotheses 105
Directional and Nondirectional Alternative Hypotheses 106
What Are Parametric and Nonparametric Tests of Statistical Significance? 108
Parametric Tests 108
Nonparametric Tests 109
What Is Statistical Significance? 109
Errors Made in Interpretations of the Significance Level 111
What Are Type I and Type II Errors and Power? 111
Type I Errors 112
Type II Errors 113
Relative Seriousness of Type I or Type II Errors 113
Power 114
Five Methods of Increasing Power 114
What Are the Types of Statistical Significance Testing Methods? 117
Parametric Tests of Statistical Significance 118
Nonparametric Tests of Statistical Significance 122
What Are the Different Sampling Methods? 125
Probability Sampling 126
Nonprobability Sampling 129
What Are the Sampling Decisions Researchers Make? 130
What Is Sampling Error? 132
Summary 132
Discussion Questions 134
xvi Extended Contents
5 Experimental Designs 135
Objectives 135
Overview 136
What Are True Experimental Designs? 137
Box. Research Examples 137
What Are the Most Common True Experimental Designs? 138
Pretest–Posttest Control‑Group Design 138
Posttest‑Only Control‑Group Design 142
Solomon Four‑Group Design 144
What Are Factorial Experimental Designs? 146
Analysis of Data 146
What Are Quasi‑Experimental Designs? 150
What Are the Common Quasi‑Experimental Designs? 150
Static‑Group Comparison Design 150
Nonequivalent Control‑Group Design 153
Counterbalanced Designs 154
Time‑Series Designs 155
What Are Preexperimental Designs? 156
What Are the Types of Preexperimental Designs? 156
One‑Shot Case Study 157
One‑Group Pretest–Posttest Design 157
When Should Researchers Use Each Experimental Research Design? 159
Summary 160
Discussion Questions 162
Illustrative Example. The Effects of Learning Strategy Instruction
on the Completion of Job Applications by Students with Learning Disabilities
by J. Ron Nelson, Deborah J. Smith, and John M. Dodd 163
Illustrative Example Questions 171
Additional Research Examples 171
Form. Threats to Internal Validity 172
Form. Threats to External Validity 174
6 Causal–Comparative Research 176
Objectives 176
Overview 176
What Is the Causal–Comparative Research Method? 177
Box. Research Examples 177
What Should Researchers Consider When Designing
a Causal–Comparative Study? 178
Development of Hypotheses 178
Selection of Groups 179
Analysis of Data 180
Can the Causal–Comparative Approach Be Combined
with Factorial Designs? 184
Classification Variables 185
Mixed Factors 185
Analysis of Data 185
When Should Researchers Use the Causal–Comparative Research Design? 186
Summary 187
Discussion Questions 188
Illustrative Example. Literate Language Features in Spoken Narratives
of Children with Typical Language and Children with Language Impairments
by Kellie S. Greenhalgh and Carol J. Strong 188
Illustrative Example Questions 202
Additional Research Examples 202
Extended Contents xvii
Form. Threats to Internal Validity 203
Form. Threats to External Validity 205
7 Correlational Research 207
Objectives 207
Overview 207
Box. Research Examples 208
What Is the Correlational Research Method? 208
What Are the Issues in Designing a Correlational Study? 208
Critical Issues in Correlational Research 208
What Are the Statistical Procedures Used in Correlational Research? 209
Important Concepts 210
Correlation Coefficient 212
Basic Correlational Procedures 220
Advanced Correlational Procedures 222
Analysis of Data 232
When Should Researchers Use the Correlational Research Method? 234
Summary 235
Discussion Questions 236
Illustrative Example. Variables That Influence Decoding and Spelling in Beginning
Readers by Kathy Strattman and Barbara Williams Hodson 237
Illustrative Example Questions 250
Additional Research Examples 251
Form. Threats to Internal Validity 252
Form. Threats to External Validity 254
8 Survey Research Methods 256
Objectives 256
Overview 256
What Are the Purposes of Survey Research? 257
Description 257
Box. Research Example 257
Explanation 258
Exploration 258
What Are the Different Types of Surveys? 258
What Are the Factors in Choosing a Survey Method? 258
Sampling Procedures 258
Sampling Population 259
Question Format 260
Content of Questions 260
Response Rate 260
Time and Money 261
How Is Survey Research Designed? 261
Sampling 261
Developing the Survey Instrument 263
Analysis of Data 266
When Should Researchers Use Survey Research? 267
Summary 267
Discussion Questions 268
Illustrative Example. Teaching Spelling in the Primary Grades: A National Survey
of Instructional Practices and Adaptations by Steve Graham, Paul Morphy,
Karen [Link], Barbara Fink‑Chorzempa, Bruce Saddler, Susan Moran,
and Linda Mason 269
Illustrative Example Questions 287
Additional Research Examples 287
xviii Extended Contents
Form. Threats to Survey Validity 288
Form. External Validity 289
Part IV Qualitative Rese arch Methods
9 Basic Understandings in Qualitative Research 293
Objectives 293
Overview 294
What Are the Characteristics of Qualitative Research? 296
What Are the Differences between Qualitative and Quantitative Research? 298
Contextual Data 298
Descriptive Data 299
Process of Change 301
Grounded Theory Development 301
Phenomenology 303
What Are Qualitative Research Procedures? 304
Phases 304
Sampling 305
What Is “Understanding” in Qualitative Research? 308
Descriptive Validity 309
Interpretive Validity 309
Theoretical Validity 309
Generalizability 310
Evaluative Validity 311
What Are the Evaluative Criteria for Judging the Reliability and Validity
of Qualitative Research? 311
Criterion 1: Completeness of Information 312
Criterion 2: Adequacy of Interpretation 312
Criterion 3: Determination of Inconsistencies in Data 312
Criterion 4: Adequacy of Metaphors, Pictures, or Diagrams 312
Criterion 5: Collaboration with Participants 312
Criterion 6: Multiple Methods to Gather Data 312
Criterion 7: Disqualification of Interpretations 314
What Are the Types of Triangulation Methods? 315
Data Sources Triangulation 315
Analyst Triangulation 315
Theory/Perspective Triangulation 316
Methods Triangulation 316
How Are Qualitative Data Analyzed? 317
Judgment in Qualitative Research 318
Internal Validity 318
External Validity 319
Analyzing Qualitative Research 319
Computer Software for Qualitative Research 320
Summary 321
Discussion Questions 322
10 Data Collection and Designs in Qualitative Research 323
Objectives 323
Overview 324
Box. Research Examples 324
What Are Field‑Oriented Studies? 324
Case Study 324
Observation Studies 325
Box. Field Notes 326
